Note: Deprecated!

An introduction

In short, a flow is a template definition of recognizable data. It is used to transform a stream of neurons into another stream of neurons (doesn’t sound very useful, but trust me, it is).  To get a visual image on how a flow works, you can perhaps think of those toddler toys where the child needs to fit different types of blocks into different types of wholes (see image).

Well, a Flow is the yellow filter at the top of the box. As the blocks pass along, a flow tries to find a set that fits it’s filter. When it finds one, it collects this set, tags it, and stores it for further use. This process is done by the flow recognition algorithm.

A flow is a template definition, used to recognize sequences of data in a stream.

flow items

So, what is this filter that’s supposed to recognize the data, actually constructed of? Well, flow definitions are very (mmm, perhaps not so very) similar to Coco/R definitions. In short, you use constants, other (nested) flows, loops and options to build up a definition. Here’s a short example:
This flow can recognize the sequence: Agent verb possibly followed by a not. It’s actually the definition for the simple past tense (there’s an extra hidden filter on ‘verb’ which I’m not showing yet for this example). ‘Agent’ is underscored, indicating that it is another flow, verb is not, so it’s a static, like not. Which is surrounded by strait brackets, indicating that it’s optional. So basically, this flow can recognize statements like: I swam, you dug, the brown men found not,… Without the filter (like here), sentences like: I fish, they eat not,… would also be recognized.

Statics and nested flows

Most of the items in a flow definition tend to be statics and other, nested flows. This forms the meat, the data that can be found in the input stream itself. When 2 statics are declared in sequence, the same sequence needs to be found in the input, without anything in between.

Nested flows can easily be recognized. They are underlined. If you double-click on them, you will jump automatically to the flow (you can navigate back and forward between the selected flows using the navigation commands). Behavior-wise, nested flows are the same as statics. If you declare 2 nested flows in sequence, the input stream should contain the data for the 2 flows in the same sequence.

Floating flows

A floating flow is a special type of flow that can be recognized anywhere in the stream without having to be specifically declared in any other flows. They are mostly used for things like spaces and newlines. You usually don’t integrate them into other flows, as nested.

Floating flows can be destructive or not. When a destructive flow is found, it will break up the recognition of other data in the stream. So this type of flow can not be found between 2 statics, but can be between 2 nested flows or between options, loops or conditional parts (see further). If you need to have a floating flow between 2 statics, it needs to be non destructive. The type of flow can be assigned from a context menu on the flows: you can select for normal flows (not floating), floating (visualized by a green bar in front of the flow) and non destructive floating (displayed using a blue bar).

You can also specify if the data that a floating flow finds is discarded or not. Discarding the data can be useful in cases where you are really not interested in it, like for empty space. This is also selected from the context menu on the flows and is visualized by using an extra bar at the end of the flow.

Options

With an option you can define a number of different paths, from which 1 can or has to be selected. The following example of options defines the ‘can’ verb:A blue block (also called ‘conditional part’) represents a single path. If the frontal bracket of the options contains a vertical line, 1 path has to be selected, if there is no line, no path has to be selected but 1 can be. The content of a conditional has to be recognized in the same sequence as the way that it is defined and can consist of statics, flows, loops and other options, but no conditional parts. The option itself can only contain conditional parts.

Just like statics and nested flows, you can also reuse the same option in different places. In fact, the more you can do this, the more processing power can be saved. The same goes for the conditional parts (the blue blocks). These can also be shared by different options (and loops). This can be done by drag and drop from within the editor.

Loops

Loops are very similar to options, they also consist of conditional parts and they can also be defined with a vertical line in front, indicating that 1 path has to be selected or not. The main difference with options is that a loop can recognize 0, 1 or more of it’s conditional parts in sequence. They are declared using curly brackets, like in the following example: This loop simply declares a sequence of digits, with at least 1 required digit. In other words, it’s the definition for an integer.

Loops can have some more lines in the front to declare extra info. If there is a green line present, to the right of the black one, it means there can’t be a floating flow in between 2 conditional parts, otherwise this is allowed (see floating flows). A red line to the left of the black one, indicates exactly the opposite: 2 conditionals need to have a floating flow in between them for a valid parse (not often used).

As a final example, here’s the entire scanner definition which is used to recognize words, numbers (integer and doubles), signs and spaces:

Note: Deprecated!

The designer is a strange beast: the project that is opened for editing, is also running in the background as a neural network. This means that you can use the input/output channels just like in a regular (neural network enabled) application, like the AICI client app. Grate for debugging and trying out stuff, but you don’t want to get your test data mixed into the actual network, enter sandboxes:

A sandbox is a complete copy of your project, running in a new, separate designer.

So, after you have opened the Aici project in the designer (installed at {Documents}\NND\Demos\AICI_1, note that you need to select the directory, not the file), you can run your project in a sandbox. Simply click on the button in the toolbar, or select ‘debug/sandbox’ from the menu. This will start a new designer with you sandbox project loaded. Once the sandbox is open, you can use the already declared text-channel to communicate with the network. if not yet opened, switch it on by selecting ‘view/communication channels/Text channel’ from the menu. The Echo words demo has more info on this type of channels (though the view has been updated a bit). here’s the short version:

The toolbar:
1: Send the input text to the network.
2: a combo for selecting the method to send text to the network.
3: copy the conversation log to the clipboard.
4: save the conversation log to file.
5: clean the conversation log.
6: toggle audio on/off.
The edit area:
1: All the neurons that were sent to the network as input.
2: All the neurons that were sent back from the network as output.
3: The conversation log.
4: The input box

In short, simply type some text in the textbox at the bottom, press enter and watch what happens.

When you are done testing, simply close the sandbox app and you are done (it wont ask to save any changes). If you would want to do some more testing later on with the same data, you can first save it to a different location (otherwise the sandbox data will be deleted when you start a new sandbox).

For instance, the abstract can be ‘a house’ while the concrete is ‘The house that I live in,… my house. Aici needs a way to distinguish the 2 and be able to find relationships between both. This is done through the use of different data structures.

Objects

Lets start with the objects, these are the simplest. They represent abstract knowledge. Every interpretation of every word is represented by an object. For instance, ‘house’ can mean the house that you live in, but could also refer to a musical style. So there is an object for each interpretations of the word. Furthermore some words can have synonyms, words that have the same meaning. For instance, ‘house’ and ‘home’ could be considered as synonyms, or house and ‘house music’. With ‘house music’ being a compound word: 2 words joined together to form a new meaning. Other examples of compound words are: car factory, fire hose, film studio,…

Another bit of information that can be useful to know about a word, is how it should be interpreted in the context of a sentence: can it be a noun, verb, adjective,…. Sometimes, a word can have multiple interpretations, while being used as the same sentence type (like house), sometimes multiple sentence types are allowed for a single meaning (colors for instance can be used as adjectives or nouns: ‘my eyes are blue’, ‘that blue is pretty’ ). As a speed optimization, we group all the objects together that have the same part of speech, but with different meanings in ‘Pos-groups’, though this is not required.

As a visual example, let’s take the objects ‘hello’ and ‘goodbye’ (as in a greeting). Both are nouns (it’s an hello and a goodbye). Both have multiple synonyms: hi, howdy, bye,..  So here’s how this would look like:

Relationships between objects: the Thesaurus

Objects are stored in a thesaurus like structure to indicate relationships between them. For instance, both ‘aici’ and ‘jan’ are names, so  this could be expressed in a thesaurus relationship like in the 2 images below:

Notice (to the left) that the ‘noun’ filter was selected in the thesaurus together with the ‘is a’ relationship. To the right, you can see how this is represented internally: Name points to an ‘is a’ cluster, which contains all the related objects. So the ‘Noun filter works by only displaying objects that point to ‘noun’ or are clustered by a ‘noun’ pos-group while the ‘is a’ relationship will filter out all but the clusters attached to the roots using this ‘is a’ neuron.

Sometimes a root object doesn’t have any children (as in cluster-children, not in the thesaurus). This makes the object a dummy, a placeholder that shouldn’t be used for input or output directly, but only for filtering. Some examples of these dummies can be seen when the ‘pronouns’ are selected.

The thesaurus relationships can be used in frames to filter the input in a general way and to perform lookups by the actions. For instance, the ‘be name’ frame (in the ‘names’ editor) has an ‘object flow’ element that filters on ‘name’ through the ‘is a’ relationship, as seen in the image below.

This means that the frame is only selected when the input contains the result of an object flow that is the child of an ‘is a’ cluster, attached to ‘name’ or one of it’s thesaurus children (so multiple levels are allowed).

To create this type of filter by the way, you simply drag ‘name’ from the thesaurus to the filter area.

Actions also make use of these thesaurus relationships in a very similar way. For instance, an action could check if an object  is allowed to/can be the name of something (‘my name is Jan’). Or, in sentences like: ‘this is blue’, the thesaurus is used to find out what ‘blue’ actually is: a color (I’m jumping here, keep with me, It will become clear very soon).

An object can also be the child of multiple items in a thesaurus. For instance, ‘blue’ is both the child of ‘colorful’ and ‘sad’ (child of ‘emotional’). Note that these are all adjectives, hence we say ‘colorful’ instead of ‘color’. To find out which relationship to take, we can make use of a log of previously made statements or a list of focused objects. The most important usage of either color(ful) or emotion(al) in this log, triggers it for the new input. Most important can be interpreted very broadly: it could be ‘newest log item’, or ‘most talked about’,… If it is impossible to solve this, a question needs to be asked.

Anyway, once you have figured out which path to take in the thesaurus: colorful, we still have an adjective. If the same logical concept/idea can be expressed in multiple parts of speech, using different words (like color-colorful) and you want to be able to jump between the 2, you need to have a relationship between them. As I already stated, we interpret ‘blue’ as a ‘color’, not ‘colorful’. Why, will become more clear when we talk about the assets, for now lets just take it that we need to change from adjective or verb to noun. As you can see in the image, the thesaurus has a special section to define this type of relationship: the left part of the table contains the link meaning, the right section is the ‘from’ part of the link.

The same can be done for verb conjugations. In English, a verb usually has 3 different conjugational forms, other languages have others, so the thesaurus allows you to define this type of relationship free form, much like the part of speech relationships. When you create a new verb though, the designer will try to fill in all the conjugations (if it finds neurons titled ‘present particle’, ‘past particle’ and ‘third person present’. It’s always best to check them before storing the default though, since it doesn’t know about irregular verbs and sometimes it/I may simply goof up.

The verb conjugations, compared to the part-of-speech relationships, are used a bit differently. These serve their purpose while parsing the input and generating output. Some flow elements try to search for conjugation relationships while filtering (in the ‘FilterCode’ tab) in order to get the correct parse. For instance, in the ‘model verbs flow’, this is used to make certain that a verb is the present particle. The same goes for generating output: when a verb is used in certain situations (ex: ‘what’ contains a verb instead of a noun), we need to generate it’s present particle form (or another, depending on the exact situation).

Assets

Up until now, all relationships were between objects, so between abstract knowledge only. To record a network’s experiences (the data that it records), we use assets and asset relationships.

At it’s core, an asset is a cluster with meaning ‘asset’ (we’re very original here), which contains a number of child neurons that represent property-value pairs for that asset. This is done through links to objects and/or other assets, using the meanings: attribute, value, amount,… From this basic structure, we can start to play and create meaningful relationships.

As usual, a picture says a thousand words, so to the left, I’ve displayed a debug view of the ‘text channel’ neuron (the text input channel of AICI). Why this neuron? If you look closely, it has a link that points to an ‘asset’ cluster, using the meaning ‘Pr:You’. So with this image, I’ve got 2 birds with 1 stone: the asset and how to find the asset that AICI is using for the person on the other side of the channel .

A text channel stores a reference to the asset that represents the entity it is communicating with using a link with meaning ‘Pr:you.

As a side note, linking ‘you’ to the text channel, allows the system to talk to multiple users at the same time: each on their own channel.

So what’s stored in this asset anyway? As you can see, there are 2 child neurons in the asset, so 2 data items: the first has an attribute (property) of ‘entity’ and a (property) value of ‘animate entity’. The second asset-child  points to ‘eye’ for the attribute, the value is another asset and it has a general amount of ‘plural’.

Note that the attribute should always be a noun. Having a fixed transformation point facilitates the search.  Why a noun: well, all adjectives and action-verbs can be converted to a noun, but not the other way round. This conversion can be done using the part of speech relationships that were previously described.

The ‘entity’ property is how AICI tries to classify assets: is it an object, animate, animal, human,… If you look at the thesaurus (in the designer), ‘entity’ is a noun-root and the asset-child’s value is one of it’s children. This is a rule: when the value is an object, it must have a thesaurus relationship with the attribute. This type of child asset represents the ‘x is an y’ type of relationship. Here, it is ‘I am an animate entity’, an other example could be:  ‘I am a human’, which would/should map to the same asset-child.

- An asset-child with an object as value represents an ‘x is an y’ sentence structure.
- In this case, the value must have a thesaurus relationship with the attribute. This relationship can span multiple levels.

AICI at the moment,  presumes that an ‘entity’ that is able to communicate through a text channel, must be animate (mammal, AI,…). So, even if this information is not yet provided, it is presumed to be the case. This is done cause the property and value play a role in other parts of the network.

The second asset-child has another asset as value. This type of relationships represents the ‘x has an y’ sentence structure. More specifically, in this example we said: ‘I have blue eyes’. This also explains the undefined ‘plural’ amount: eyes is plural and was converted into a single value, but since we don’t know exactly how many, we keep a ref to the multiple. The sub asset also has it’s own child-asset, which defines the ‘color-blue’ part of the example sentence. Through this nesting, it is possible to describe complex entities that consist out of multiple parts of knowledge.

An asset-child that references another asset as value, represents an ‘x has an y’ sentence structure.

Objects referencing assets

Assets aren’t exactly islands, independent of everything else. No, they too are referenced in many different ways. We already saw 2: assets attached to child-assets, and to the text-channels.  There’s one more reference worth mentioning at this moment: objects that reference assets. Yes, assets don’t just reference objects, it can also go the other way round. This is used to represent general knowledge about general things, like: ‘a human has hands’, as depicted in the image to the right. The same thing is also valid for statements like ‘an x is a y’: they are represented as assets, linked from objects.

Note: I did a bit of recycling with regards to the meaning of the link from the object: it also uses ‘asset’, which is the same as the meaning for the cluster.

An object that references an asset cluster, using ‘asset’ as meaning for the link, is used to represent statements of the form: ‘an x is/has a y’

The important difference between stand alone assets and assets that are linked to objects, are the information that they represent about the agent of a sentence: stand alone assets represent concrete agents: I, you, the book,… while object referenced assets represent abstract agents: a book, a human,…

These 2 data structures: objects and assets, are simple, but, I believe, flexible enough to represent a whole world of information. It is not as much the the querying and reshaping of the data as it will, in the end, be the automatic creation of these structures which I believe will ultimately be the truly interesting thing. With a query, we can retrieve data, reshaping it allows us to find truths, fallacies and falsities in the data, but a correct automatic creation of a new, complex asset, is in fact, a new entity.

In the end, all is just structure.

Note: Deprecated!

The thunk to .Net is finally working:

So it’s now possible to call static functions that take value types as argument and/or as return type.  In this example, I am calling the standard System.IO.File.Copy function to copy a file on my disk. Exceptions are also supported, as you can see from the first line, when I am trying to copy a non existing file. The output comes from the exception generated by the copy. Note that you can specify the arguments in different ways. In this example, I am using the ‘to’ word. ‘From’ would also be recognized. Both determine the extraction order of the arguments.

At the moment, I have hardcoded this extraction of the function arguments for the ‘copy-verb’ manually. The idea is to change it to something more general that can work for any .net function using some meta data of the function (like which neuron to send to which output, how to recognize the arguments, how many arguments,…). I’m not there yet though. First some finishing touches to the designer, so I can trace some neuron/memory leaks. After that I’ll probably do a new release, so you can play a bit.

Oh, and it’s still way to hot outside

Note: Deprecated!

No, this post doesn’t explain what AICI is, check here to read more about that topic. Today, I’m attempting to explain how to add the functionality to AICI so that it can respond to the statement ‘What’.

As previously mentioned, there are a number of stages that we need to get through to go from input to output. The first is the scan phase, which translates the letters into words, numbers and signs. When this is done, we try to find sentence constructs, like the subject, verb,…. This is matched to a frame, which determines the action that needs to be triggered. Finally, the action might also activate an attribute, but this is out of this post’s scope. So let’s make our network respond to ‘what’.

Recap:
First check if the flows can parse the statement: both scanner and grammar. Next comes the frame and finally the action and attribute.

Scanner

This stage will transform the letters into a single word ‘what’, when it reaches the ‘Sentence’ flow. The result of this stage is a cluster, with the meaning ‘Sentence’ (the same as the name of the flow that produced the result). The child will be a textneuron, representing ‘what’

‘What’ is a simple word, which is already handled by the scanner, so no worries there. If you are wondering how words are parsed, take a look at the preceding image, which is the flow for a word. Underlined words are references to other flows,  words that aren’t underlined represent static values. Curley brackets mean a repetition, strait ones define an option. If a conditional has a vertical line after it’s first bracket, a selection is required, this is valid for both repetitions and loops. This basic syntax was inspired by parser generator languages like Coco/R, Lex and Yacc.

So in the above example, a word can start with number (a sub flow), letters or capital letters. A word must always have at least 1 letter or capital, may have middle numbers and can end with a number or an alpha numeric.

Note: at the time of writing, there is still a little optimization required to get words with many mixed digits and letters working properly.

Grammar

Moving on to the grammar section. This is also defined in a flow editor, called ‘English grammar’. So open this (in the ‘Flows’ folder) and immediately select the ‘Interrogative pronouns’ (if you are, like me, not that very fluent in English grammar, and could use a mental aid now and then, there are plenty of resources on the net, check out Google dictionary, the English club, Wikipedia,…). As you can see, our ‘what’ is already in the ‘Interrogative pronouns’ flow.  So perhaps no extra work for the grammar stage as well? As it turns out, this will be the case. For those who want some more details though, here goes:

If we  look more closely to the flow name itself (on the first picture), we can see there is an overlay ‘FL’, which is short for flow-code. If you hover over the ‘FL’, you get a tooltip with more info. This says ‘References code that is executed when the item is recognized in a data flow’. This means that the flow does something when it is recognized in an input stream. Usually, it formats the recognized data in such a way that the next stage knows what type of sentence part it was. If there is no ‘FL’ overlay, it means that the data will be recognized in the input stream, but no extra actions are taken, when found. So the input is simply copied over to the result which leaves it unrecognizable. The ‘Reflective pronouns’ are a good example of a flow without overlays.

But what to do in such a case? Before you start to code, first see who uses the flow anyway, maybe they do something with it.

Tip:
Use F8 to browse all relationships of a neuron.

If you select the ‘Reflective pronouns’ Flow (click 2 times, the first time we automatically focus on the editor) and press F8, you can see which clusters it belongs to (and some other stuff too, not important for now). In this case it’s only 1 cluster: a conditional part (we need flows), so if we drill down a bit further, until we eventually get to the ‘Object flow’. This is used to find the object of a sentence. These flows do have an overlay (in the editor), and tag the recognized data. Usually, this is enough. Only when you need to tag this sub result as well, do you need to add some code, which is generally not the case.

Anyway, to get back to our ‘Interrogative pronouns’, lets see what the attached code does, and open the code editor (right-mouse button on the flow, and select ‘View code’). Most often, as with this flow, it simply duplicates the content of the ‘Result’ variable (which always references a cluster at this stage), and removes all outgoing links on this duplicate (sometimes, result clusters can link to some extra data which we don’t want,  but is used by the algorithm) and stores this duplicate back into the result cluster.

To better explain what this actually means, lets take the following input stream: ‘you are here’. After the scan stage, we have a result that looks like image 5: a cluster with meaning ‘sentence’ and 3 text neurons as children. Once the grammar stage is done transforming this dataset, it looks more like image 6: a cluster with meaning ‘statement’  and 3 children again, but with some more info. The first 2 represent the agent (sentence subject) and the verb, both flows use the previously described transformation code. The last child is the object representation of the textneuron ‘here’. It is the result of an adverb that was recognized somewhere in a flow, which didn’t transform the data, so content simply got copied to the final result.

AICI:
When a flow has been recognized, the ‘flow-code’ callback of the flow is executed. The ‘Result’ variable will contain a cluster with the flow as it’s meaning and the recognized data as it’s children.

AICI:
Use the ‘TransformFlowResultCallback’ cluster as a default callback for flows so the result can be recognized by the frames.

Frames

Up until now, not much had to be done, with the frames though, this will change, here we actually need to do something to get it working. Frames can be seen as the conduit between the front (scanner/grammar) and back (actions/attributes). A frame defines which sentence parts it can contain and their order. Multiple orders can be defined, so that you can have the same sentence parts to form a statement or a question. You can also have optional parts, to make frames even more flexible. Each order is defined in a frame sequence. This is also able to render itself back as output, completing the circle.

Just like for the scanner and grammar, the network is already capable to extract the correct frame sequence, based on the set of already defined frames and their sequences (so if it’s not yet defined, the network can’t recognize it). Once, the sequence is found, the framework performs some calls to attached code that you have to provide so that the network also knows what to do with the input statement. The following code blocks can/should be attached to a frame sequence:

• ExtractAction: this code block is called to determine which action(s) the frame should trigger. Multiple actions are allowed, in which case each will be triggered in the order as they were returned. This is done through the variable ‘ExtractedActions’.  The attachment is required.
• ConvToKnowledge: this code block should transform the result data of the grammar parse, into the input data for the actions and return this cluster through the variable ‘KnowledgeConv’. (edit 11/July/2010) The attachment is required. This attachment is optional. If you define a value for the new column ‘ResultType’ in the frame editor and the ‘role’ neuron has a ‘ConvToKnowledge’ code cluster, the network can automatically generate the result cluster for an entire frame sequence, and so this code is no longer required. You can override the auto rendering by defining this code cluster.
• RenderCode: This code block is called by an action to render output according to the structure of the frame. The output to render is found in the ‘CurrentFrom’ system variable. The format of the data is the same as that of ‘ConvToKnowledge’. The attachment should only be defined if there is an action able to call it is output. This is not always the case, for instance, frames that deal with ‘unknown’ word types, don’t need to be generated.

Recap:
A frame determines the action to execute, provides the data for the action and is able to render data provided by an action.

As previously mentioned, a frame needs to prepare the data for the action. This should be a cluster without meaning (it will automatically be assigned, once the correct frame sequence is found, and will eventually indicate that the cluster is an input statement). The content of this cluster needs to be well defined, so that the action can handle it correctly. In theory, you can use any type of structure that you want, as long as the action uses the same. In practice, it’s best to use a predetermined structure, shared by all the actions and frames, for easy sharing of default actions. So here’s a small overview of the possible content:

• The first child neuron should indicate the time value: current, passed or future. Though most actions don’t specifically expect it on the first pos, all current frames use this.
• The child list can also contain a ‘not’ neuron, to indicate that the statement should be interpreted as a negation.
• The sentence agent should be stored in a cluster with meaning ‘who’
• You can store the sentence object in a cluster with meaning ‘what’ or ‘value’, depending on the situation. Sometimes, ‘what’ can be extracted, from ‘who’, a previous statement or the ‘value’.
• a cluster with meaning ‘verb’ can be used for sentences that a verb. Originally, this wasn’t done for the basic verbs like ‘to be, ‘to have’, since it can be inferred from the action that is triggered. However, it can always be useful to take the actual verb that was used to the action, in case it needs to do some analysis on the ‘subject’ indicator of the word. For instance, ‘am’ is usually for ‘I’.

Frames can play a little bit with these structures: fill in missing data, switch who, what and value,… Though most of the code in the current frame sequences is shared: if there is an agent in the sentence, 9 out of 10 times it is transformed in the same way, so lots of copy paste (or drag drop) possibilities here. Also perhaps for future automatic learning.
(Update 11/July/2010) In the new version, it is no longer required to provide this code for each frame. It is moved to the level of the ‘role’ (as defined in the frame element), usually these are the flows like ‘agent’ or ‘be verb’,…

The translation results for the input ‘i am jan’. In the final stage, ‘what’ is filled in by the frame.

For our example, we will have to create a new frame to handle the input ‘what’. Since this is a sentence that only contains an interrogative pronoun, we should probably best put it in the ‘Pronouns’ frame editor together with all the other pronouns (for easy lookup later on). So open up this editor (double click on the editor’s name in the project view), or create a new one if it doesn’t yet exist (the button that looks like a rectangle, you know ‘frame’).

To create a new frame, click on the button with the square, on the editor (tooltip: Create a new frame) or press Ctrl+Shift+F and provide a name for the new frame. Lets call it ‘what frame’. To quickly build the frame, you can drag neurons from the toolbox, thesaurus, flow,… and drop them on the frame editor. To start with, find an ‘Interrogative pronoun (to get it in the explorer, press ‘F4’),  and drop it somewhere in the top part to create a new frame element. This indicates that the sentence should contain an ‘interrogative pronoun’ sentence part.

Tip:
To quickly sync the explorer with the currently selected item, press F4 from anywhere in the application.

Make this element an evoker. Only evoker elements are used to find all possible frame candidates. This is a small optimization for the search algorithm. Each frame should have 1 evoker element. Usually, this is the verb, but in this case there isn’t one so we use what’s available.

At this point, we have a frame that will be activated for all sentences that contain an interrogative pronoun, like: what, who, where, when,… We need to further specify that we only want to respond to ‘what’. This is done by adding a filter to the frame element. So make certain that the element is selected, and from the thesaurus, drag ‘what’ to the bottom part of the frame editor and drop it there. this will automatically create a thesaurus filter for the currently selected relationship. So if ‘what’ had one or more children in the thesaurus, these would also be valid for the filter. In this case we don’t have that, so the relationship field could have been left empty.

Also notice the ‘RestrictionMofidierInclude’ in front of the filter. This is to indicate that input values that contain the filter value are allowed. If this was set to ‘RestrictionModifierExclude’, only sentences that wouldn’t contain ‘what’ would be allowed (so, where, who, when,..).

Finally, there is also the small button. When this is checked, the content of the sentence part should only contain what is defined in the filter, and no other words. If not checked, other words may still be present. This is an important little property which I almost got wrong starting out this post. Do we want to handle sentences like ‘what color’, ‘what time’, what the hell’,  ‘what in the name of sweet Jesus’, or only ‘what’. If you go back to the very first image, you can see that it’s allowed to put some words after the pronoun. This is to allow correct recognition of sentences like ‘what size are you’ where ‘you’  is the subject. This means that the grammar will also recognize ‘what color’ as an interrogative pronoun. It turns out this is actually ok for us (see frame sequence code). So we leave this not checked to allow other words in the flow result.

The Frame sequence

Now that the frame has been defined, lets move on to the sequence. To create this, press the ‘Add frame sequence button’, or ‘Shift+Ctrl+S’ and provide a name for the sequence (make certain that the frame editor is active when pressing the button, otherwise it won’t be active). Lets call it ‘What statement’. You create the sequence by moving the frame elements from the left part to the right part of the editor. You can use the arrow buttons for this. Multiple elements can be moved at the same time. The up and down can be used to change the order.

All pretty strait forward. The cool part comes in the code editor. To open this, select the frame sequence in the editor (or somewhere else, like in the explorer), open the context menu and select ‘view code’. First, lets make certain that this new code editor is also available from the project. All you have to do is open the context menu on the bottom tab items of the code editor, and press the ‘Show in project’ menu item. This will add the neuron to the currently selected project folder (or root if there is non selected). So, when you do this, it’s best that the correct folder is selected first. If you forgot to select the one you wanted, you can always still move it later on.

Tip:
To add a code editor to the project view, right click on the bottom tab items and select ‘Show in project’. The item will be added to the currently selected project folder.

By default, there are only a limited number of pages available in the code editor. The rules and actions pages are always available (though the code clusters are only created when you actually drop something on them). But not the ones that we need. Those will have to be added manually. So open the context menu, again on the bottom tab items, and select the required pages listed under ‘add new code page’. We need ‘ExractAction’, ‘ConvToKnowledge’ and ‘RenderCode’.

Tip:
You can add any type of code page to a code editor, make certain that the required neuron is tagged as ‘a default meaning neuron’ in the explorer. Open the context menu on the bottom tab items of the code editor and select the new page from the ‘Add new page’ menu item.

Frame sequence code

RenderCode: The render code for the ‘what’ statement as displayed in the above picture is very simple, but not completely correct (see below), we basically write “what?”. I added the question mark, just to show a little trick of the text sins: When you send them multiple neurons, by default, each one will be rendered on it’s own line. To group neurons together, use the ‘BeginTextBlock’ and ‘EndTextBlock’ neurons.  These are recognized by the text sin as special formatting tokens.

We use the ‘output’ statement to send data to a sensory interface. The fastest way to declare this expression, is by dragging it from the toolbox. This has 2 pages: one for common expressions like clusters, assignments and commonly used neurons, and a second page, exclusively for all the instructions.

Tip:
You can quickly expand or collapse all toolbox items from their context menu. All instructions display a tooltip with a short description, the expected parameters and output.

The first argument to all the output statements is always the same: ‘OutputSin’. This is a global that is filled in by the framework and points to the sensory interface to which the output should be sent. You can also hardcode which sensory interface it needs to be sent to, but that is less flexible. This way, you can actually have multiple text-sins in 1 network and have multiple users chat at the same time. As a small side note: because the output statement can take any number of arguments, we could have written it shorter: in 1 statement. This wasn’t done because I forgot, before taking the screenshot

Also, if you look closely, some of the statements appear to have a double border, while others don’t. This is to indicate that it is used multiple times. This is important to keep in mind, if you modify these statements, you are also changing them in some other part of the code. When dragging code around, it is not like text that you duplicate. When you drag and drop, there is no deep copy, only a new reference made. This can keep the code size pretty compact, allows for some deep future analysis and most importantly should allow for easy auto generation later on. But it is something to keep in mind while changing code.

Caution:
Statements with a double border are used multiple times, be careful while modifying these.

The complete render code looks something more like the following image. This is able to handle not only ‘what’, but also ‘what color’, ‘what on earth’,… Looks scary? Don’t worry, it’s actually very simple, and most of the code is copied over from other parts anyway.

The second statement, the big yellow monster, is actually a query, who’s result is stored in the ‘what’ variable. ‘Get children filtered’ returns all the child neurons of a cluster that meet a specified filter. In this case, we only want neuron clusters that have a cluster-meaning equal to ‘sg:What’, in other words, we want to retrieve the ‘what’ part from the input. (no surprise there).
The next statement is an ‘if’ construct: if we found something (count(what) > 0), render the contents of the ‘what’ variable, using the ‘Render object’ code block. If there is no cluster found, simply render ‘what?’. The ‘what’ variable isn’t passed over to the ‘Render object’. This is not possible and not required: a code block doesn’t have it’s own variable space, so all the variables maintain their value, hence, the ‘what’ variable is passed along implicitly.

And thus, you have mastered your third little peace of code already

ConvToKnowledge: (update 11/July/2010: this code cluster is no longer required, but is still allowed) The transformation for ‘what’ is also fairly strait forward. This  type of attached code tends to have a fairly similar structure across frame sequences. In this one, we check if the interrogative pronoun contained anything other than ‘what’. We do this by checking if it has more than 1 child. When this is the case, we create a new cluster, assign it the meaning ‘sg:what’ (this is a tag operation) and we copy all the children, except the first, from the cluster that we found to the newly created one. Finally, we add this new cluster to our result. We don’t copy the first child, since that is the the ‘what’. We presume this, if you know any situation in which it is not the case, the code needs to be updated to something more secure (remove the actual ‘what’ neuron).

ExtractAction: Lastly, we also need to provide some code that returns the action we want to execute. Before we can do this, we actually need to know which are the possible actions that can be performed.

So lets try and come up with a few possible dialogs, so we can analyze the different possibilities. Here’s what I came up with on short notice:

PC: I have hair
You: what
PC: I have hair –> this is a simple repetition of what was last said.

PC: I am in the black room
You: what color
PC: black-> return the first value that was found through an ‘is a’ relationship (black is a color), from any of the words of the prev statement.
You what room
PC: the room of your house –> return more relationship info on the actual result of ‘the black room’, if there is any. In this case, we know the owner of the house.

You: what the fuck
PC: No swearing please –> catch a special case.

So Basically, I have 3 different groups of the conversations (if you can think of more, le me know). In the first case, when the user simply asks ‘what’, we return the last output statement. The 3th conversation, when the user wants to express surprise/disagreement through an insult, we return a standard message to keep it clean. Finally, in the middle example, some analysis is done on the previous statements, to find a proper match. We can check the actual words, see if there is an ‘is a’ relationship to be found. In this  case, the user might have forgotten the actual value, but still remembered the attribute. The user could also be requesting more info about something, in which case we need to return some more links if there are any.

Exactly how you pour this information into actions, is a bit up to you. Me, I would go for 3 different actions. In fact, I would handle the last conversation type (aka: ‘what the xxx’) as a special case in a different frame, so we can filter on these specific word groups. I guess that would be a good exercise for you. The first and second should be handled by this frame, so these are 2 new actions that should be returned by our frame sequence. To determine the exact action that needs to be returned, we can use a similar ‘if’ construct as in the ‘ConvToKnowledge’ code cluster: when there is more than a ‘what’ in the input data, use ‘Find Prev out statement’, which does the heavy searching. If there is only ‘what’ in the input data, use the light weight ‘Return last out statement’. To return the action we want to execute, simply assign it to the ‘ExtractActions’ variable, like in the statements to the right.

Recap:
Return your actions in the ‘ExtractedActions’ variable.

Actions

Rejoice, the end of this post is almost in sight: Actions, the final step. As already mentioned, we are going to need 2 actions, though to keep the post a bit within the limits of sanity, I am going to restrict the explanations to the simplest action. The more complex search algorithm needs another one of these posts to explain it all. In fact, I still have to finish it myself. Perhaps, I’ll leave it like it is, as an exercise for you

Before we start to create our action though, perhaps a small word about actions in general. These are actually nothing more then regular neurons with 2 different types of code clusters attached. The first type contains code that is called when the action is executed. You can specify a different code cluster for all the different states that the action needs specific responses for. The default state is ‘Normal state’, which is currently the only one defined, but you can create and set as many of your own states as you want. The current state of the network is defined as a link between the text sin using the meaning ‘state’ to the current value (see image). This means that you can have as many concurrent network states as communication channels. Also, when an action doesn’t have an implementation for the current state, or when the text-sin doesn’t define a state, the fallback ‘Normal state’ is used.

The second type of code clusters that are attached to actions, are used during the conversation, to check for and handle possible responses to questions. That is, some actions require some sort of response. These types of actions need to check if a response is valid, determine if an action is a valid response action and can also specify extra ‘response’ code. This though is again far outside the scope of this post, so lets leave  it at that.

Back to our example, we still need to create the action. Since this is just a neuron, there are many different ways of doing this. If you are already in the correct project folder, a quick way is to use the toolbar button above the project overview, called ‘Create a new neuron with attached code’. This will create a neuron, open it’s code editor and add it to the project. If you press F2, you can change the name (make certain the project node has keyboard focus). To add the ‘Normal state’ code page, just do the same as with the frame sequences: open the context menu on the bottom tab items, select ‘add new code page’ and click on ‘Normal state’.

And that’s how the action should eventually look like. It’s a nice bit of code to end with, with just a little bit more meat to look at. Though in the end, it’s still very strait forward. Before we delve into the specifics of the code, perhaps first a small word about the conversation log, since that’s our primary data source in this action.

The conversation log is a cluster, attached to the text sin for which it is keeping track of the previous statements made by the network and the user of the text sin. This means that the network has a log cluster for each active user. It keeps track of a 1 to 1 conversation. A Part of the content of the log consists out of the clusters that we created in the frame sequence, in the ‘ConvToKnowledge’ code. These have a cluster-meaning of ‘in’. The other part consists out of the clusters that were generated in the actions, as output.  These are labeled ‘out’. Both ‘in’ and ‘out’ children still have a link to the action(s) that was/were executed so we know what they triggered.

When we translate this information to our action, we get something like the previous picture. Basically, we try to retrieve the log from the ‘OutputSin’ global. We have already encountered this global.  It is the one that is always pointing to the Text-sin for whom we are currently processing, remember. If this doesn’t have a log, something is wrong, so we show a message to the user, log it as an error (this is useful for the debugger) and exit the link. We exit the link and not the neuron or the entire processor, since there might still be other actions that need to be executed.
If there is a log file, we walk through each log entry, from back to front, using a counter and the ‘GetChildAt’ instructions. You can do this in different ways (using the reverse instruction combined with 1 or more selects, for instance). I choose this way, since I think it’s the fastest solution for this situation.
If we find an item, with a cluster meaning of ‘out’, we simply push it back on the stack and exit the loop. By putting the cluster back on the stack, we actually let the network execute the action again, since it is still linked to the data. Note that we decrease the counter each time that we don’t have an item with meaning ‘out’. The ‘-‘ instruction might be written a bit strange at first. You can probably best think of it as a polish notation. This is actually a result of the fact that ‘-‘ is just an instruction and at present, all instructions have the same UI input format. And that’s it.

Ok, slowly breed in, and out. You’ve just reached the top of the Himalayas (or something comparable). It’s done, you’ve absorbed all the info, and your still alive. Time now to let it all breed a little, let it digest, and sleep on it a bit. Don’t worry about it, it will come to you too.

Note: Deprecated!

I have created a new small network host demo application which you can use instead of the full designer to run AICI. You can get it from here, or from the new download section.

It’s basically a general purpose network host that only supports the text sensory interfaces for input/output. You specify the network that needs to be run at the command prompt. The installation program will do a default setup to work with the AICI demo, but you can change the shortcut to use your own if you want.

Note: Deprecated!

The new release is finally ready.  The Aici demo took a bit longer than planned. Also, lots of things have been fixed and updated. Here’s a non exhaustive list:

• There’s a complete new lockmanager running in the background. This is much more secure (thread-wise, that is) and a lot faster. It’s still a bit of a diesel though, it takes some time for it to get going, but once running, it should be pretty fast. The slow start is due to the storage mechanism (all xml files currently). This is the major drag on the entire system at the moment, and will be fixed next.
• Wordnet import has been seriously updated, a lot more info is retrieved, and it’s now also possible to import the entire db in one go (although not yet advisable, due to a memory bug in the designer, it still takes a major byte out of the hard disk and it simply takes ridiculously long).
• The thesaurus has been given a make over to allow for editing and filtering. He can now also display / edit non recursive relationships. Drag drop is also supported.
• The frame editor has been updated considerably: drag drop support has been added and frame element filters/restrictions have also been introduced (will probably be extended in the future).
• I changed the function of the ‘contains’ operator a bit. It now only checks the contents of a variable. For clusters/children, there are the new instructions (IsClustesteredBy, LinkExists, ContainsChildren, GetInFiltered,…).
• I have added the ‘not contains’ operator.
• New instructions:
• Arithmetic group (+-/*%): I finally caved on those. My original plan was to see how far I got without using any arithmetic in the neural code. I guess, this is as far as I got with that.
• Get-at group: get child at, get cluster at, get out at, get in at, get info at.
• Distinct
• get Incoming, get outgoing, get info, Get in filtered, get out filtered, Get info filtered
• Is clustered by, Link exists, Contains children
• Remove-at group: Remove child at, remove info at, remove link in at, remove link out at
• many, many bug fixes, updates and little improvements.

Aici 1

This demo is a small chat interface. Though, in it’s current state, it’s not much more than a framework. It can initiate a conversation, close it, ask for the name of the user and store the data. It’s not yet able to fully recognize a recurring user since I haven’t defined the neural code for this yet.

I will be explaining how it works and how you can expand on this functionality, shortly. For those who can’t wait and want to get a peek under the hood, here are some pointers to get started:

• There are 3 main stages:
• Flow recognition, which is basically the syntactical stage: check the word types and order. Project pages are:
• the flows (aici/flows),
• the code that is attached to these flows (aici/code/flow code)
• the code that recognizes the flows in the input (aici/code/flow recognition)
• Frame recognition, or the semantics stage. This is where we try to find meaning in the words. Project pages are:
• the frames (aici/frames)
• the code that is attached to the frame sequences (aici/code/frame seq code). The frames don’t have code (yet).
• the code that recognizes the frames in the flow results (aici/code/frame recognition).
• action execution, or the response of the network to the input. Project pages are:
• Actions: all the different actions that the system knows (not yet a lot, should be extended).
• Output: some common code blocks for rendering output. This is also used by the frame sequences, since they are also used to render data in a predefined format.
• Action helpers: code that the action neurons can use to perform common tasks, like stopping a conversation or controlling the timers. Timer callbacks are also stored here.
• The transition between the different stages can be located in aici/code/transition.
• More specifically, the ‘Transition’ neuron is used to go from the flows to the frames and finally to the actions. 
• An action is started using the ‘Execute action’ neuron as meaning for a link from a data cluster to the action that needs to be started.
• The project also contains some mindmaps that describe the inner data structures and functionality.

Note: Deprecated!

Intro

Time for the second demo overview: the Scanner.  It builds on most of the ideas found in the first demo but it goes way further, and actually does something very useful (although you wouldn’t say it at first).  It’s probably going to be a lengthy piece so I’m thinking of cutting it in 2 or maybe even 3 parts. Anyway, lets first start it up, either through the start menu shortcut (in the Demo’s sub folder, conveniently called Scanner demo), or by opening it in NND (File/Open, select the ‘My documents/NND/Demos/Scanner’ folder). Once the project is loaded, you should see a single text communication channel open (called Text sin), if this is not the case, go to View/Communication channels/Text sin and make certain that it is is selected.

Overview

Lets get a taste of what it does, so enter some text (or leave the one that’s already there) and press the ‘send’ button (or enter).

You’ll notice that it basically does the same thing as the echo demo: the input text is echoed back, except that it’s a bit slower. If you had the debugger tab open, you probably also noticed a lot more activity, so something more must be going on.

And indeed, if you take a closer look to the text sin channel, in the upper section, you can see the neurons that were send to the network (on the left) and those that were returned (on the right), which are different. This was not the case with the echo demo, it simply sent all the incoming neurons back out, as they were. In this demo though, we get back something completely different: TextNeurons, that represent the same thing as the int neurons that were sent as input (if you regard them as ASCII characters). Hence the name of the demo, it’s a scanner.

Converting a stream of ASCII chars into words, integers, doubles and signs is a pretty useful feature and that’s all this demo is capable of doing, but that’s only because I stopped there. You see, in the background, is a general purpose algorithm that converts an input stream of neurons into a single result cluster, using any and all of the flows that are defined in the network. This means that you can use the same algorithm with many different flow definitions. I have simply defined some to recognize words, integers and doubles. You could go further and add flows to find verbs, sentence subjects,.. (in fact, that’s what the AICI 1 demo does). You could even go further still and create flows for visual objects or audio fragments, the same algorithm can be used. Unfortunately though, the editor doesn’t yet support such types of displays for flows (will probably be added somewhere in the future though).

Details

So how is the translation actually performed? To explain this, let me first recap some of the basic concepts of neurons and flows:

• The different types of  data available to a neuron are: incoming and outgoing links, possibly one or more parent clusters, for clusters possibly 1 or more children and a meaning. And finally value neurons also have their value of course. This is important, cause when the translation process starts, this is all the available information.
• Flows are nothing more than clusters that contain flow items, which can be statics or conditionals (loops and options). These in turn can only contain conditional parts. They represent a single branch of the decision tree. Parts can again have the same data as flows: statics or conditionals. So if you are a neuron (a static, part, conditional or flow), you can always look up into your list of parents to see in which flows and parts it is used.

Now, if you recall from the first demo, an input starts by creating an IntNeuron for each ASCII value, which is linked to another neuron using the ‘Letter’ neuron (ID 109). These are all put on the execution stack and the processor starts (the Rules code on the Letter neuron is executed).

So both neurons are new and only have each other as links. This means that we can’t use links or parent/child relationships to resolve the first step, but instead must use something different. The only other thing that remains is the value of the int neurons, so this is compared against some constants to see if they are digits (0..9) alpha numeric (a..z+A..Z), spaces | returns, or something else.  This comparison results in the creation of 1 new neuron per input neuron: the result cluster, in which we store the integer. One of these clusters (or it’s duplicate, due to a split) will eventually store the end result.  This cluster is linked to one of 3 static neurons: Digit, Alpha or Space (signs like . or , are handled a bit differently, this will be explained later). Naturally, if the integers would represent color values, we would use other starting points than digit or alpha. In other words, this first part is variable according to the type of input and the required accuracy of the algorithm. As meaning, we use  the start of the ‘flow recognition’ algorithm, called ‘Stage 1.1’ and put the result cluster back on the stack.  It’s important to put this one back on the stack, and not the item we are looking for. That’s because the algorithm can perform numerous splits and we want the result to be duplicated not the searchable, cause the contents of the list are continuously modified and we don’t want the result of one processor to be modified by  another one (I had to learn this the hard way).

After this initial step, the actual recognition algorithm kicks in. This consists out of 4 stages, grouped by 2. Meaning that  stage 1.2 is executed immediately after stage 1.1 for each neuron (this is the same for stage 2,1 and 2.2), but at the end of stage 1.2 and 2.2 all the result neurons are collected into a single global. Only after the last link of the last item on the stack has  been processed, are all the result clusters put back on the stack, with links for the next stage.  This is done for allowing to group items together.

The different stages are:

• Stage 1.1 (search parts/flows): Find conditional parts or flows in the list of parents of the searchable. If there are multiple results, perform a split for each, after the result list has been filtered (these are the shortcuts). If there are no results and this is the only and last item still on the stack, the end result has been found.
• Stage 1.2 (sequence-combine and filter): Check if items are sequential (2 flow items declared after each other in the same parent list, which is a part or flow) and handle floating flows, which are allowed to appear anywhere in the input stream, but which break up the sequence of other items.  Different actions can be performed if the order of the items is not ok: try to solve further or exit without result. Results of sequential items are grouped together.
• Stage 2.1 (search conditionals): Find conditionals in the list of parents of the searchable if this is a conditional part, otherwise the stage is simply skipped. If there are multiple results, perform a split for each after the result list has been filtered (not yet completely implemented at this stage). If there are no results, there is an error in the flow definition.
• Stage 2.2 (process loops and sync-points): If there was a conditional found in the previous stage, check if this is a loop. If so, and the previous item is of the same loop, combine the results. Also start a sync-point (will be explained later) if this was defined on the conditional.

These 4 stages are repeated until there is only 1 result cluster on the stack that represents the end result of a flow which is no longer used in any other flows. This cluster is made the result of the  split for the processor it ran on.  Off course, because there were possibly many splits, there could be many results. These will all be presented in the split-callback cluster (you need to provide a code cluster to the Split instruction, which will be called when all sub processors are done). In this demo, the result is sent back to the sin that caused the input, in a normal situation thought, this will simply start another process, as is done in the AICI 1 demo.

During this whole process, the algorithm is capable of executing callback code (attached to the statics, conditionals, parts and flows) at certain specific moments in the code. This is where the magic happens. The following types of callbacks are possible (together with  their execution time):

• Flow code: this code cluster is executed when a flow has been recognized in the stream. The ‘Result’ variable (ID 1822) contains all the neurons that match the flow. It’s here for instance that the int neurons are converted to a single word using the CiToS (Cluster with ints to string) instruction.
• Filter flow code: this code cluster is called from stage 1.1  (or 2.1, but this is not yet completely implemented) just after all the next items were retrieved from the list of parents of the searchable (stored in the CurrentTo variable). It allows the flow item to determine if it is a valid result, given the current state of the network. This is done by checking the contents of a number of globals, like ‘Prev stage item’ (ID 1956), which contains the previously processed neuron.

In the next post, I’ll go deeper into the specifics of the algorithm itself, for as you’ve guessed by know, it’s a bit funky, and I don’t want to forget all the subtleties, since it’s definitely still a work in progress (there are many improvements still possible).