1/* Part of SWI-Prolog 2 3 Author: Jan Wielemaker 4 E-mail: J.Wielemaker@vu.nl 5 WWW: http://www.swi-prolog.org 6 Copyright (c) 2007-2016, University of Amsterdam 7 VU University Amsterdam 8 All rights reserved. 9 10 Redistribution and use in source and binary forms, with or without 11 modification, are permitted provided that the following conditions 12 are met: 13 14 1. Redistributions of source code must retain the above copyright 15 notice, this list of conditions and the following disclaimer. 16 17 2. Redistributions in binary form must reproduce the above copyright 18 notice, this list of conditions and the following disclaimer in 19 the documentation and/or other materials provided with the 20 distribution. 21 22 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 23 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 24 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 25 FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 26 COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 27 INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, 28 BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 29 LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 30 CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 31 LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN 32 ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 33 POSSIBILITY OF SUCH DAMAGE. 34*/ 35 36:- module(http_stream, 37 [ http_chunked_open/3, % +Stream, -DataStream, +Options 38 http_is_chunked/1, % +DataStream 39 http_chunked_flush/2, % +DataStream, +Extensions 40 http_chunked_add_trailer/3, % +DataStream, +Key, +Value 41 stream_range_open/3, % +Stream, -DataStream, +Options 42 multipart_open/3, % +Stream, +DataStream, +Options) 43 multipart_open_next/1, % +DataStream 44 45 % CGI Stream interaction 46 cgi_open/4, % +Stream, -DataStream, :Hook, +Options 47 cgi_property/2, % +Stream, -Property 48 cgi_set/2, % +Stream, -Property 49 cgi_discard/1, % +Stream 50 is_cgi_stream/1, % +Stream 51 cgi_statistics/1 % ?Statistics 52 ]). 53:- use_foreign_library(foreign(http_stream)). 54:- public http_stream_debug/1. % set debug level 55 56:- meta_predicate 57 stream_range_open(,,). % onclose option is module sensitive 58 59/** <module> HTTP Streams 60 61This module realises encoding and decoding filters, implemented as 62Prolog streams that read/write to an underlying stream. This allows for 63sequences of streams acting as an in-process pipeline. 64 65The predicate http_chunked_open/3 realises encoding and decoding of the 66HTTP _Chunked_ encoding. This encoding is an obligatory part of the HTTP 671.1 specification. Messages are split into chunks, each preceeded by the 68length of the chunk. Chunked encoding allows sending messages over a 69serial link (typically a TCP/IP stream) for which the reader knows when 70the message is ended. Unlike standard HTTP though, the sender does not 71need to know the message length in advance. The protocol allows for 72sending short chunks. This is supported totally transparent using a 73flush on the output stream. 74 75The predicate stream_range_open/3 handles the Content-length on an input 76stream for handlers that are designed to process an entire file. The 77filtering stream claims end-of-file after reading a specified number of 78bytes, dispite the fact that the underlying stream may be longer. 79 80@see The HTTP 1.1 protocol http://www.w3.org/Protocols/rfc2616/rfc2616.html 81*/ 82 83%! http_chunked_open(+RawStream, -DataStream, +Options) is det. 84% 85% Create a stream to realise HTTP chunked encoding or decoding. The 86% technique is similar to library(zlib), using a Prolog stream as a 87% filter on another streQam. Options: 88% 89% - close_parent(+Bool) 90% If `true` (default `false`), the parent stream is closed 91% if DataStream is closed. 92% 93% - max_chunk_size(+PosInt) 94% Define the maximum size of a chunk. Default is the default 95% buffer size of fully buffered streams (4096). Larger values may 96% improve throughput. It is also allowed to use 97% set_stream(DataStream, buffer(line)) on the data stream to 98% get line-buffered output. See set_stream/2 for details. 99% Switching buffering to `false` is supported. 100% 101% Here is example code to write a chunked data to a stream 102% 103% ``` 104% http_chunked_open(Out, S, []), 105% format(S, 'Hello world~n', []), 106% close(S). 107% ``` 108% 109% If a stream is known to contain chunked data, we can extract 110% this data using 111% 112% ``` 113% http_chunked_open(In, S, []), 114% read_stream_to_codes(S, Codes), 115% close(S). 116% ``` 117% 118% The chunked protocol allows for two types of _out of band_ data. 119% Each chunk may be associated with additional metadata. That is 120% achieved using http_chunked_flush/2. The last chunk may be followed 121% by HTTP header lines. That can be achieved by calling 122% http_chunked_add_trailer/3 before closing the chunked stream. 123% 124% After http_chunked_open/3, the encoding of DataStream is the same as 125% the encoding of RawStream, while the encoding of RawStream is 126% =octet=, the only value allowed for HTTP chunked streams. Closing 127% the DataStream restores the old encoding on RawStream. 128% 129% @error io_error(read, Stream) where the message context provides 130% an indication of the problem. This error is raised if 131% the input is not valid HTTP chunked data. 132 133%! http_is_chunked(+DataStream) is semidet. 134% 135% True if DataStream is created using http_chunked_open/3. 136 137%! http_chunked_flush(+DataStream, +Extensions) is det. 138% 139% Emits the next chunk flush_output/1 on DataStream, but in addition 140% adds extension parameters to the chunk. Extensions is a list of 141% `Key=Value` terms. For example, to close a chunked stream with an 142% error chunk we something like below. First, we flush the last 143% pending data, next we fill a new chunk and flush it with 144% extensions. 145% 146% ``` 147% flush_output(current_output), 148% format("Sorry, something went wrong!\n"), 149% http_chunked_flush(current_output, [error=true]) 150% ``` 151% 152% @compat It turns out that most clients ignore chunked extensions. 153 154%! http_chunked_add_trailer(+DataStream, +Key:atom, +Value:atom) is det. 155% 156% Add a trailer key/value to DataStream. If the stream is closed, 157% each call adds a line ``Key: Value\r\n`` to the output. The 158% strings for Key and Value need to be compliant with the HTTP 159% header syntax. 160% 161% @compat It turns out that most clients ignore trailer lines. The 162% JavaScript fetch() method should make these available as 163% ``response.trailer''. 164 165 166 /******************************* 167 * RANGES * 168 *******************************/ 169 170%! stream_range_open(+RawStream, -DataStream, +Options) is det. 171% 172% DataStream is a stream whose size is defined by the option 173% size(ContentLength). Closing DataStream does not close 174% RawStream. Options processed: 175% 176% - size(+Bytes) 177% Number of bytes represented by the main stream. 178% - onclose(:Closure) 179% Calls call(Closure, RawStream, BytesLeft) when DataStream is 180% closed. BytesLeft is the number of bytes of the range stream 181% that have *not* been read, i.e., 0 (zero) if all data has been 182% read from the stream when the range is closed. This was 183% introduced for supporting Keep-alive in http_open/3 to 184% reschedule the original stream for a new request if the data 185% of the previous request was processed. 186 187 188 /******************************* 189 * MULTIPART * 190 *******************************/ 191 192%! multipart_open(+Stream, -DataSttream, +Options) is det. 193% 194% DataStream is a stream that signals `end_of_file` if the 195% multipart _boundary_ is encountered. The stream can be reset to 196% read the next part using multipart_open_next/1. Options: 197% 198% - close_parent(+Boolean) 199% Close Stream if DataStream is closed. 200% - boundary(+Text) 201% Define the boundary string. Text is an atom, string, code or 202% character list. 203% 204% All parts of a multipart input can be read using the following 205% skeleton: 206% 207% == 208% process_multipart(Stream) :- 209% multipart_open(Stream, DataStream, [boundary(...)]), 210% process_parts(DataStream). 211% 212% process_parts(DataStream) :- 213% process_part(DataStream), 214% ( multipart_open_next(DataStream) 215% -> process_parts(DataStream) 216% ; close(DataStream) 217% ). 218% == 219% 220% @license The multipart parser contains code licensed under the 221% MIT license, based on _node-formidable_ by Felix Geisendoerfer 222% and Igor Afonov. 223 224%! multipart_open_next(+DataStream) is semidet. 225% 226% Prepare DataStream to read the next part from the multipart 227% input data. Succeeds if a next part exists and fails if the last 228% part was processed. Note that it is mandatory to read each part 229% up to the end_of_file. 230 231 232 /******************************* 233 * CGI SUPPORT * 234 *******************************/ 235 236%! cgi_open(+OutStream, -CGIStream, :Hook, +Options) is det. 237% 238% Process CGI output. OutStream is normally the socket returning 239% data to the HTTP client. CGIStream is the stream the (Prolog) 240% code writes to. The CGIStream provides the following functions: 241% 242% * At the end of the header, it calls Hook using 243% call(Hook, header, Stream), where Stream is a stream holding 244% the buffered header. 245% 246% * If the stream is closed, it calls Hook using 247% call(Hook, data, Stream), where Stream holds the buffered 248% data. 249% 250% The stream calls Hook, adding the event and CGIStream to the 251% closure. Defined events are: 252% 253% * header 254% Called if the header is complete. Typically it uses 255% cgi_property/2 to extract the collected header and combines 256% these with the request and policies to decide on encoding, 257% transfer-encoding, connection parameters and the complete 258% header (as a Prolog term). Typically it uses cgi_set/2 to 259% associate these with the stream. 260% 261% * send_header 262% Called if the HTTP header must be sent. This is immediately 263% after setting the transfer encoding to =chunked= or when the 264% CGI stream is closed. Typically it requests the current 265% header, optionally the content-length and sends the header 266% to the original (client) stream. 267% 268% * close 269% Called from close/1 on the CGI stream after everything is 270% complete. 271% 272% The predicates cgi_property/2 and cgi_set/2 can be used to 273% control the stream and store status info. Terms are stored as 274% Prolog records and can thus be transferred between threads. 275 276%! cgi_property(+CGIStream, ?Property) is det. 277% 278% Inquire the status of the CGI stream. Defined properties are: 279% 280% * request(-Term) 281% The original request 282% * header(-Term) 283% Term is the header term as registered using cgi_set/2 284% * client(-Stream) 285% Stream is the original output stream used to create 286% this stream. 287% * thread(-ThreadID) 288% ThreadID is the identifier of the `owning thread' 289% * transfer_encoding(-Tranfer) 290% One of =chunked= or =none=. 291% * connection(-Connection) 292% One of =Keep-Alive= or =close= 293% * content_length(-ContentLength) 294% Total byte-size of the content. Available in the close 295% handler if the transfer_encoding is =none=. 296% * header_codes(-Codes) 297% Codes represents the header collected. Available in the 298% header handler. 299% * state(-State) 300% One of =header=, =data= or =discarded= 301% * id(-ID) 302% Request sequence number. This number is guaranteed to be 303% unique. 304 305%! cgi_set(+CGIStream, ?Property) is det. 306% 307% Change one of the properies. Supported properties are: 308% 309% * request(+Term) 310% Associate a request to the stream. 311% * header(+Term) 312% Register a reply header. This header is normally retrieved 313% from the =send_header= hook to send the reply header to the 314% client. 315% * connection(-Connection) 316% One of =Keep-Alive= or =close=. 317% * transfer_encoding(-Tranfer) 318% One of =chunked= or =none=. Initially set to =none=. When 319% switching to =chunked= from the =header= hook, it calls the 320% =send_header= hook and if there is data queed this is send 321% as first chunk. Each subsequent write to the CGI stream 322% emits a chunk. The implementation does __not__ use the 323% chunked stream filter defined by http_chunked_open/3. It 324% shares most of the implementation though and CGI streams 325% do support http_is_chunked/1, http_chunked_flush/2 and 326% http_chunked_add_trailer/3. 327 328%! cgi_discard(+CGIStream) is det. 329% 330% Discard content produced sofar. It sets the state property to 331% =discarded=, causing close to omit the writing the data. This 332% must be used for an alternate output (e.g. an error page) if the 333% page generator fails. 334 335%! is_cgi_stream(+Stream) is semidet. 336% 337% True if Stream is a CGI stream created using cgi_open/4. 338 339:- multifile 340 http:encoding_filter/3, % +Encoding, +In0, -In 341 http:current_transfer_encoding/1. % ?Encoding 342 343:- public 344 http:encoding_filter/3, 345 http:current_transfer_encoding/1. 346 347%! http:encoding_filter(+Encoding, +In0, -In) is semidet. 348% 349% Install a filter to deal with =chunked= encoded messages. Used 350% by library(http_open). 351 352httpencoding_filter(chunked, In0, In) :- 353 http_chunked_open(In0, In, 354 [ close_parent(true) 355 ]). 356 357%! http:current_transfer_encoding(?Encoding) is semidet. 358% 359% True if Encoding is supported. Used by library(http_open). 360 361httpcurrent_transfer_encoding(chunked). 362 363%! cgi_statistics(?Term) 364% 365% Return statistics on the CGI stream subsystem. Currently defined 366% statistics are: 367% 368% * requests(-Integer) 369% Total number of requests processed 370% * bytes_sent(-Integer) 371% Total number of bytes sent. 372 373cgi_statistics(requests(Requests)) :- 374 cgi_statistics_(Requests, _). 375cgi_statistics(bytes_sent(Bytes)) :- 376 cgi_statistics_(_, Bytes)