How to: Server-Sent Events with Indy HTTP Server (part 3)

In this part, the server application uses the Indy HTTP server uses SSE to continuously send events to the JavaScript EventSource.


Part 3: the demo application, now streaming

Ingredient #1: the HTML page with JavaScript

The script has not changed, it reads two data items from the ping event:

  • a time stamp in ISO 8601 format
  • the peer data, which is the IP address and port number
<!DOCTYPE html>
<html>
	<head>
		<title>SSE example</title>
	</head>
	<body>
		<script>
		const evtSource = new EventSource("sse");
		
		evtSource.addEventListener("ping", (event) => {
  const newElement = document.createElement("li");
  const eventList = document.getElementById("list");
  const time = JSON.parse(event.data).time;
  const peer = JSON.parse(event.data).peer;
  newElement.textContent = `ping at ${time} from ${peer}`;
  eventList.appendChild(newElement);
});

		</script>
		<ul id="list">

		</ul>
	</body>
</html>

The code is based on the article Using server-sent events on MDN: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events

Ingredient #2: server side code

The TIdHTTPServer subclass contains a method to provide client-specific data in the /sse resource. When invoked, it will return a single ping event with a JSON payload, containing the peer IP and port number, and a timestamp:

  function TMySSEServer.BuildContentText(
    AContext: TIdContext): string;
  begin
    Result := 'event: ping' + #13
      + Format('data: {"time": "%s", "peer": "%s:%d"}',
      [DateToISO8601(Now, False),
      AContext.Binding.PeerIP,
      AContext.Binding.PeerPort]) + #13#13;
  end; 

The DoCommandGet method uses the BuildContentText function to provide the event data, and simulates work by sleeping for a random time interval.

Note

  • The data stream is running in an endless loop (repeat ... until false).
  • Because the method never terminates, the method calls AResponseInfo.WriteHeader to send the HTTP headers to the client (line 13).
  • Neither ContentText nor ContentStream can be used to send data to the client. Instead, the event data must be sent by using the Write.. methods of the connection’s IOHandler (line 16).
 procedure TMySSEServer.DoCommandGet(AContext: TIdContext;
    ARequestInfo: TIdHTTPRequestInfo;
    AResponseInfo: TIdHTTPResponseInfo);
  var
    Data: string;
  begin
    AResponseInfo.CharSet := 'UTF-8';
    if ARequestInfo.Document = '/sse' then
    begin
      AResponseInfo.ContentType := 'text/event-stream';
      AResponseInfo.CacheControl := 'no-store';
      AResponseInfo.ContentLength := -2;
      AResponseInfo.WriteHeader;
      repeat
        Data := BuildContentText(AContext);
        AContext.Connection.IOHandler.Write(Data);
        Sleep(Random(1000));
      until False;
    end
    else
    begin
      AResponseInfo.ContentType := 'text/html';
      AResponseInfo.ContentStream := TFileStream.Create('index.html', fmOpenRead);
    end;
  end;

Output

When the browser navigates to http://localhost, the server will provide the HTML and the embedded JavaScript will start reading event data from http://localhost/sse:

Notable difference from the previous version:

  • The server sends a continuous stream of events as response to the HTTP GET request to the /sse resource.
  • The length of the response is unknown (it is virtually unlimited), therefore the HTTP response must not contain a content-length header.
  • The connection will not be closed after sending one or more events.
  • The client will only retry (reconnect and send a new request), if the the server disconnects its end of the connection, or no data is received and the connection times out.

Diagnostics

To see the full response of the server to the GET request, you may use

curl -v -N localhost/sse

Example:

Updates

Example project source code is now on GitHub

The complete code for all three projects is now available on GitHub at https://github.com/michaelJustin/indy-snippets

Example project for Daraja HTTP Framework

There is a new example project in the develop branch of Daraja. URL: https://github.com/michaelJustin/daraja-framework/tree/develop/demo/server-sent%20events

Advertisement

How to: Server-Sent Events with Indy HTTP Server (part 2)

In this second part, a server application uses the Indy HTTP server to provide a HTML page which uses SSE to update its content with data sent from the server.

Part 2: the basic demo application, some client data added

Ingredient #1: the HTML page with JavaScript

The script now reads two data items from the ping event:

  • the time stamp, now sent from the server in proper ISO 8601 format
  • the peer data, which is the remote ip address and port number
<!DOCTYPE html>
<html>
	<head>
		<title>SSE example</title>
	</head>
	<body>
		<script>
		const evtSource = new EventSource("sse");
		
		evtSource.addEventListener("ping", (event) => {
  const newElement = document.createElement("li");
  const eventList = document.getElementById("list");
  const time = JSON.parse(event.data).time;
  const peer = JSON.parse(event.data).peer;
  newElement.textContent = `ping at ${time} from ${peer}`;
  eventList.appendChild(newElement);
});

		</script>
		<ul id="list">

		</ul>
	</body>
</html>

The code is based on the article Using server-sent events on MDN

https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events

Ingredient #2: server side code

The TIdHTTPServer subclass now contains a private method to provide client-specific data in the /sse resource.

  function TMySSEServer.BuildContentText(AContext: TIdContext): string;
  begin
    Result := '';
    repeat
      Result := Result + 'event: ping' + #13 +
        Format('data: {"time": "%s", "peer": "%s:%d"}',
        [DateToISO8601(Now, False), AContext.Binding.PeerIP,
        AContext.Binding.PeerPort]) + #13#13;
      Sleep(100);
    until Random < 0.8;
  end; 

The DoCommandGet method uses the BuildContentText function to provide the event data:

  procedure TMySSEServer.DoCommandGet(AContext: TIdContext;
    ARequestInfo: TIdHTTPRequestInfo;
    AResponseInfo: TIdHTTPResponseInfo);
  begin
    if ARequestInfo.Document = '/sse' then
    begin
      AResponseInfo.ContentType := 'text/event-stream';
      AResponseInfo.CacheControl := 'no-store';
      AResponseInfo.ContentText := BuildContentText(AContext);
    end
    else
    begin
      AResponseInfo.ContentType := 'text/html';
      AResponseInfo.ContentStream :=
        TFileStream.Create('index.html', fmOpenRead);
    end;
    AResponseInfo.CharSet := 'UTF-8';
  end;

Output

When the browser navigates to http://localhost, the server will provide the HTML and the embedded JavaScript will start reading data from the address http://localhost/sse and receive one or more events.

The ping event, which the server sends to the browser, now includes the server time in ISO 8601 format and the peer IP address and port.

Next part

In the next part, the data stream will be sent continuously.

How to: Server-Sent Events with Indy HTTP Server (part 1)

In this article, a server application uses the Indy HTTP server to provide a HTML page which uses SSE to update its content with data sent from the server.

Server-Sent Events (SSE) is a server push technology enabling a client to receive automatic updates from a server via an HTTP connection, and describes how servers can initiate data transmission towards clients once an initial client connection has been established.

Server-sent events, https://en.wikipedia.org/w/index.php?title=Server-sent_events&oldid=1093881969 (last visited Sept. 17, 2022).

Part 1: a very basic demo application

Ingredient #1: a HTML page with Javascript

<!DOCTYPE html>
<html>
	<head>
	<title>SSE example</title>
	</head>
	<body>
	<script>
	const evtSource = new EventSource("sse");
		
	evtSource.addEventListener("ping", (event) => {
	  const newElement = document.createElement("li");
	  const eventList = document.getElementById("list");
	  const time = JSON.parse(event.data).time;
	  newElement.textContent = `ping at ${time}`;
	  eventList.appendChild(newElement);
	});
	</script>
	<ul id="list">
	</ul>
	</body>
</html>

The code is based on the article Using server-sent events on MDN

https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events

Ingredient #2: server side code

In the example application, a TIdHTTPServer subclass is used to deliver the HTML document when a browser accesses it.

Note that the server has to keep connections alive, so the Server property KeepAlive must be set to true:

  procedure Test;
  begin
    Server := TMySSEServer.Create;
    try
      Server.KeepAlive := True;
      Server.Startup;
      ReadLn;
    finally
      Server.Free;
    end;
  end;

The DoCommandGet method is overriden and looks like this:

  procedure TMySSEServer.DoCommandGet(AContext: TIdContext;
    ARequestInfo: TIdHTTPRequestInfo; AResponseInfo: TIdHTTPResponseInfo);
  var
    S: TStream;
    C: string;
  begin
    if ARequestInfo.Document = '/sse' then
    begin
       AResponseInfo.ContentType := 'text/event-stream';
       AResponseInfo.CacheControl := 'no-store';
       AResponseInfo.CharSet := 'UTF-8';
      
         C := 'event: ping' + #13
           + Format('data: {"time": "%d"}', [GetTickCount]) 
           + #13#13;

       AResponseInfo.ContentText := C;
       AResponseInfo.ResponseNo := 200;
    end
    else
    begin
      S := TFileStream.Create('index.html', fmOpenRead);
      AResponseInfo.ContentType := 'text/html';
      AResponseInfo.ContentStream := S;
      AResponseInfo.ResponseNo := 200;
    end;
  end;

Output

When the browser navigates to http://localhost, the server will provide the HTML and the embedded JavaScript will start reading data from the address http://localhost/sse (and receive only one event). As specified in the HTML spec for EventSource, the client will retry after some seconds:

Each EventSource object has the following associated with it:
– A reconnection time, in milliseconds. This must initially be an implementation-defined value, probably in the region of a few seconds.

HTML – 9.2 Server-sent events (https://html.spec.whatwg.org/multipage/server-sent-events.html#server-sent-events)

Note: this is just the first step. In the next part, we will implement a continuos data stream.

Indy FTP LIST timeout with Delphi 11 in active mode (solved)

In the Indy issue tracker, a bug was reported which first appeared with Delphi 11.

FTP LIST timeout with latest Delphi 11 (and idFTP)

It is possible to reproduce the bug not only with Delphi 11 but also with Free Pascal. Reportedly the timeout also occurs with Delphi 10.4.

If you want to reproduce it with your version of Delphi (or Free Pascal), you may take a small test project from https://github.com/IndySockets/Indy/issues/377#issuecomment-932632490 and configure it to connect with a local or public FTP server.

Test with Delphi 2009 and Free Pascal 3.2.0 (Lazarus 2.0.12) on Windows 10

With Delphi 2009 and Free Pascal 3.2.0, I reproduced a read timeout in line 2134:

if LReadList.ContainsSocket(LDataSocket) then
begin
  LPortSv.Listen(0);
  Self.GetResponse([125, 150, 154]);  <--------------- read timeout
end else
begin
  Self.GetResponse([125, 150, 154]);
end;

STAT> Connected.
RECV> 220 Microsoft FTP Service
SENT> HOST test.rebex.net
RECV> 504 Server cannot accept argument.
SENT> USER demo
RECV> 331 Password required for demo.
SENT> PASS password
RECV> 230 User logged in.
SENT> FEAT
RECV> 211-Extended features supported:
RECV> LANG EN* UTF8 AUTH TLS;TLS-C;SSL;TLS-P; PBSZ PROT C;P; CCC HOST SIZE MDTM REST STREAM211 END
SENT> OPTS UTF8 ON
RECV> 200 OPTS UTF8 command successful – UTF8 encoding now ON.
SENT> TYPE A
RECV> 200 Type set to A.
SENT> SYST
RECV> 215 Windows_NT
SENT> TYPE A
RECV> 200 Type set to A.
SENT> PORT 192,168,178,20,204,119
RECV> 200 PORT command successful.
SENT> LIST
STAT> Disconnected.
Exception at 000000010001DCD9: EIdReadTimeout:
Read timed out.

Tested with servers

  • test.rebex.net
  • ftp.dlptest.com

Call for testers

It would be interesting to confirm if the bug only appears in some specific versions of Delphi and Free Pascal. If you can reproduce it, you may comment here.

Resolved

The issue is resolved now

How to: Let’s Encrypt certificates with Indy HTTP Server

This article guides you through the setup for a simple HTTPS server using Let’s Encrypt certificates.

Download the project source

The example project is included in the Daraja demo folder at https://github.com/michaelJustin/daraja-framework/tree/master/demo/https. You may use the project IndyHttpsTest with Delphi 6 or later, or with the Lazarus IDE. This stand-alone project only requires the Indy library, it does not depend on the Daraja HTTP framework.
Three lines in the project specifiy the domain certificate, the domain certificate key and the intermediate certificate:

...

IOHandler := TIdServerIOHandlerSSLOpenSSL.Create;
IOHandler.SSLOptions.CertFile := 'cert.pem';  // domain certificate
IOHandler.SSLOptions.KeyFile := 'key.pem';  // domain key
IOHandler.SSLOptions.RootCertFile := 'cacert.pem'; // intermediate certificate
IOHandler.SSLOptions.Mode := sslmServer;

...

Install the OpenSSL DLLs

The OpenSSL libraries can be downloaded from https://indy.fulgan.com/SSL/ (note that there are different versions for 32 and 64 bit applications). Extract the archive and copy the DLLs to the daraja-framework\demo\https folder.

Intermediate certificate

Let’s Encrypt uses an intermediate certificate which is available for download at https://letsencrypt.org/certificates/

From the Intermediate Certificates section, download the active intermediate certificate, copy it to the daraja-framework\demo\https folder, and rename it to cacert.pem

Server certificate and key

Now you need your server certificate and key files. An easy way to do this is the free ‘SSL Certificate Wizard‘ at https://zerossl.com/ – this service allows to create Let’s Encrypt certificates for your web domain.

Domain verification

Before you can download server certificate and key, the ZeroSSL certificate wizard must verify that the domain is under your control. Verification options are HTTP or DNS based. To verify domain ownership using HTTP verification, you will need to create appropriate files with specific text strings under your “<webroot>/.well-known/acme-challenge/” directory. Only after ZeroSSL checked these files, your certificate will be issued.

Finally, you need to download the files domain-crt.txt and domain-key.txt from the last page of the SSL Certificate Wizard.

Certificate installation

Copy the files domain-crt.txt and domain-key.txt to the project daraja-framework\demo\https folder with the names cert.pem and key.pem

Ready to launch

Verify the certificate chain

You may inspect the certificate properties in Windows if you copy or rename domain-crt.txt to domain.crt and double-click to open it. The certificate information window willl show the certificate properties (validity period etc.) and the certification chain:

Local test

Compile and start the IndyHttpsTest project in the folder daraja-framework\demo\https. The program will launch your default web browser and navigate to https://127.0.0.1. As the certificate is not issued for 127.0.0.1, the browser will display a security warning which displays the name mismatch.

Only if the server runs on the web server which the certificate has been issued for, browsers will accept the certificate without warnings.

Disclaimer

Please understand that this how to guide is only meant to be a short introduction into SSL/TLS certificate usage and you should always be aware of potential security risks.

Consuming Server-Sent Events (SSE) with Indy TIdHTTP and TIdEventStream

A new Indy HTTP client / JAX-RS server example is now available on GitHub. The server side generates Server-sent events. Server-sent events (SSE) is a technology enabling a browser to receive automatic updates from a server via HTTP connection.

The example code uses TIdHTTP and TIdEventStream to connect to the server, and writes the incoming events to the console window.

indy-sse-jaxrs

Requirements

  • Delphi or Lazarus IDE
  • Indy 10.6.2
  • Java IDE
  • Java EE 8 application server

Client

The SSE client setup in the TIndySSEClient.Create creates a TIdHTTP instance and a TIdEventStream instance.
As recommended, the Accept header is set to 'text/event-stream'. It also sets the Cache-Control header to 'no-store' to prevent proxy servers from caching the result.
Our event handler TIndySSEClient.MyOnWrite is assigned to the TIdEventStreamOnWrite property.

constructor TIndySSEClient.Create;
begin
  inherited Create;

  SSE_URL := URL;

  EventStream := TIdEventStream.Create;
  EventStream.OnWrite := MyOnWrite;

  IdHTTP := TIdHTTP.Create;
  IdHTTP.Request.Accept := 'text/event-stream';
  IdHTTP.Request.CacheControl := 'no-store';
end;

The OnWrite handler decodes the UTF-8 encoded event data and writes it to the console:

procedure TIndySSEClient.MyOnWrite;
begin
  WriteLn('Received ' + IntToStr(Length(ABuffer)) + ' bytes');
  WriteLn;
  WriteLn(IndyTextEncoding_UTF8.GetString(ABuffer));
  ...
end;

Server (main REST method)

The server generates the server-sent event and places a Stock instance in its data part:

  @GET
  @Path("prices")
  @Produces(MediaType.SERVER_SENT_EVENTS)
  public void getStockPrices(@Context SseEventSink sseEventSink) {
    int lastEventId = 1;

    while (running) {
      Stock stock = getNextTransaction();

      System.out.println("Send event ...");
      OutboundSseEvent sseEvent = this.eventBuilder
              .name("stock")
              .id(String.valueOf(lastEventId))
              .mediaType(MediaType.APPLICATION_JSON_TYPE)
              .data(Stock.class, stock)
              .reconnectDelay(3000)
              .comment("price change")
              .build();

      sseEventSink.send(sseEvent);
      lastEventId++;
    }
    sseEventSink.close();
  }

Additional notes

  • this is the first draft of a SSE client, which does not support features such as reconnect
  • the example assumes that every call of MyOnWrite contains exactly one event, teminated by a sequence of two line separators (cr lf / cr / lf) so that no extra code is required to do proper event stream parsing

Full source code

https://github.com/michaelJustin/indy-sse-jaxrs

References

Single file upload example using Indy TidHTTP and multipart/form-data

A new cross-language example for HTTP and REST is now available on GitHub.

Description

Single file upload example using Delphi client and Java server code and HTTP multipart/form-data

Requirements

  • Delphi 2009 or newer
  • Indy 10.6.2
  • Java JDK 8 or newer
  • Apache Maven
  • WildFly application server

Client

program IndyPostFormData;

{$APPTYPE CONSOLE}

uses
  IdHTTP, IdMultipartFormData, SysUtils;

const
  URL = 'http://localhost:8080/indy-post-formdata-1.0-SNAPSHOT/webresources/generic/pdf';

var
  Indy: TIdHTTP;
  Params: TIdMultiPartFormDataStream;
  Response: string;

begin
  Indy := TIdHTTP.Create;
  Params :=  TIdMultiPartFormDataStream.Create;
  try
    try
      Params.AddFile('file', 'client.pdf');
      Response := Indy.Post(URL, Params);
      WriteLn(Response);
    except
      on E:Exception do
        Writeln(E.Classname, ': ', E.Message);
    end;
  finally
    Params.Free;
    Indy.Free;
  end;
  ReadLn;
end.

 

Server (main REST method)

  @POST
  @Path("/pdf")
  @Consumes({MediaType.MULTIPART_FORM_DATA})
  public Response upload(MultipartFormDataInput input) {
    String UPLOAD_PATH = "c:/tmp/";
    try {
      InputStream fileInputStream = input.getFormDataPart("file", InputStream.class, null);
      String fileName = "test.pdf";

      int read;
      byte[] bytes = new byte[1024];

      try (OutputStream out = new FileOutputStream(new File(UPLOAD_PATH + fileName))) {
        while ((read = fileInputStream.read(bytes)) != -1) {
          out.write(bytes, 0, read);
        }
      }
    } catch (IOException e) {
      throw new WebApplicationException("Error while uploading file. Please try again");
    }
    return Response.ok("Data uploaded successfully").build();
  }

 

Full source code

https://github.com/michaelJustin/indy-post-formdata