Visiting the TMS lab day 2: Adding artificial intelligence to TMS WEB Core apps

There is no doubt about it that artificial intelligence is a hot topic. Wondering how this cool technology could be used by Delphi developers creating web applications, we researched how we could leverage existing technology. In the area of artificial intelligence, the Tensorflow.js library is one of the most popular and powerful libraries for use in web clients. So, we embarked on investigating how we could enable using TensorFlow.js easily from our beloved Pascal language in a TMS WEB Core web client application.

First sample: sequential prediction

This first small sample is based on creating first a model with sample data ourselves and then let the machine predict a value based on the model. The model is simply a series of points (X,Y values). After we feed the point data into the model, it is up to the model to predict the next point on a line.

In Pascal, this translates to using a class TTMSTFPredictSequentialNext that internally uses the Tensorflow model tf.keras.models.Sequential. This is made available as a TComponent with following public interface:

  TTMSTFPredictSequentialNext = class(TComponent)
  public 
    procedure createModelWithSample(xs, ys: TJSArray);
    function predictYforX(aValue: JSValue): JSValue;
  end;

To feed the model with sample data, a Javascript array of X values and Y values is created, initialized with the dots clicked on the user-interface in a TWebPaintBox. Drawing the dot on the TWebPaintBox is just like a Delphi developer would do this for years with the TCanvas in a TPaintbox:

procedure TForm2.DrawDot(X, Y: Integer);
begin
  WebPaintBox1.Canvas.Brush.Style := bsSolid;
  WebPaintBox1.Canvas.Brush.Color := clRed;
  WebPaintBox1.Canvas.Pen.Style := psSolid;
  WebPaintBox1.Canvas.Pen.Width := 2;
  WebPaintBox1.Canvas.Pen.color := clRed;
  WebPaintBox1.Canvas.Ellipse(X-4, Y-4, X+4, Y+4);
end;

Note that the Sequential model expects normalized data. This means that we need to normalize the pixel coordinates into the 0..1 space, meaning the minimum X value translates to 0, the maximum X value to 1.

Initialize the model

var
  xs,ys: TJSArray;
begin
  tfModel := TTMSTFPredictSequentialNext.create;
  xs := TJSArray.New;
  ys := TJSArray.New;

  FillData(xs, ys);
  NormalizeData(xs, ys);
  tfModel.createModelWithSample(xs, ys);
end;

Getting the predicted value

Note that here we of course need to first normalize the X value, i.e. convert it to the range 0..1 where 0 is the minimum X value and 1 is the maximum X value. The predicted Y value is normalized as well, so here we need to denormalize it, i.e. convert the value between 0..1 back to Y pixel coordinates based on the minimum and maximum Y value.

var
  predictY: single;
  X,Y: integer;
begin
  X := LastPointX + 50;
  predictY := tfModel.predictYforX(normalize(X)));
  y := denormalize(predictY);
end;

This is a sample prediction:



or you can discover how this works by directly using the online TMS WEB Core demo.

Second sample: digit identification

In the second sample, we show how existing model data can be loaded in the TensorFlow model and can be used to identify a drawn number. The model data is taken from a Tensorflow example using a MNIST database. It is loaded into the model with:

  tfModel := TTMSTFDigitIdentifier.Create;

When the model is initialized, we can load the sample data:

  tfModel.loadModelFromUrl('ModelJs/model.json');

We then allow to draw a number with the mouse on a TWebPaintBox:

procedure TForm2.WebPaintBox1MouseDown(Sender: TObject; Button: TMouseButton;
  Shift: TShiftState; X, Y: Integer);
begin
  doPaint := true;
  remX := X;
  remY := Y;
end;

procedure TForm2.WebPaintBox1MouseMove(Sender: TObject; Shift: TShiftState; X,
  Y: Integer);
begin
  if doPaint then
  begin
    WebPaintBox1.Canvas.Pen.Width := 30;
    WebPaintBox1.Canvas.Pen.Color := colorStroke;
    WebPaintBox1.Canvas.MoveTo(X, Y);
    WebPaintBox1.Canvas.LineTo(X + 30, Y);
    remX := X;
    remY := Y;
  end;
end;

procedure TForm2.WebPaintBox1MouseUp(Sender: TObject; Button: TMouseButton;
  Shift: TShiftState; X, Y: Integer);
begin
  doPaint := false;
end;

The next step is to get the image data from the TWebPaintBox, normalize its size, i.e. reduce it to a 28x28 image and call the method model.IdentifyDigit() to let the identification happen. This identifyDigit() method expects the image data as type JSImageData, defined in Pascal as TJSImageData. When it is recognized, IdentifyDigit returns a number from 0 to 9, when not, it returns -1. The returned value is set as caption of a panel.

The code used is:

var
  ImgData: TJSImageData;
  v: JSValue;
begin
  ImgData := normalizeddigit.Canvas.Context.getImageData(0, 0, 28, 28);
  v := tfModel.identifyDigit(x);
  if v = -1 then
    pnlDigit.Caption := '?'
  else
    pnlDigit.Caption := Format('%d', [v]);
end;

You can experiment with the model and code yourself by opening the TMS WEB Core web client app we created for this digit identifying app using the TensorFlow library.



Summary

The area of artificial intelligence is big. Different problems require different models and different approaches. We just presented two small samples here, but it is clear that much more can be done, not only with TensorFlow but also with other libraries. In our lab, we did the experimental work to use this in a convenient way from our beloved Pascal language for these two samples, but clearly, there is still quite some work ahead to expose all capabilities as convenient to use Pascal classes. We're curious and very interested to hear what AI problems you might want to solve and thus, with what priority and on what we should focus first.

Lab visit feedback & win!

Our team loves to hear what you think about what is brewing in the lab, how you plan to use the upcoming features, what priority our team should give to it and if you have possibly interesting and/or creative ideas to make this even more powerful for Delphi developers. To reward your interaction & feedback, we'll pick 3 blog comments on October 15 that we liked the most and first prize is a free TMS WEB Core license, the 2nd and 3rd prize is a 50% discount coupon on TMS WEB Core. Let yourself hear to increase your chances!

Meanwhile, you can go ahead and explore the new & exciting territories of web client development that become available for Delphi developers with TMS WEB Core! You can download the trial version that is generally available, go ahead with the standalone version you purchased or with TMS WEB Core and additional tools that are all included in TMS ALL-ACCESS. Note also that in October, you can still take advantage of purchasing TMS WEB Core at launch price from 295EUR for a single developer license. From Nov 1, regular pricing will be active on TMS WEB Core (395EUR for a single developer license).