1/* Part of SWI-Prolog 2 3 Author: Jan Wielemaker 4 E-mail: J.Wielemaker@vu.nl 5 WWW: http://www.swi-prolog.org 6 Copyright (c) 2002-2023, University of Amsterdam, 7 VU University Amsterdam 8 SWI-Prolog Solutions b.v. 9 All rights reserved. 10 11 Redistribution and use in source and binary forms, with or without 12 modification, are permitted provided that the following conditions 13 are met: 14 15 1. Redistributions of source code must retain the above copyright 16 notice, this list of conditions and the following disclaimer. 17 18 2. Redistributions in binary form must reproduce the above copyright 19 notice, this list of conditions and the following disclaimer in 20 the documentation and/or other materials provided with the 21 distribution. 22 23 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 24 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 25 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 26 FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 27 COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 28 INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, 29 BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 30 LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 31 CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 32 LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN 33 ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 34 POSSIBILITY OF SUCH DAMAGE. 35*/ 36 37:- module(http_client, 38 [ http_get/3, % +URL, -Reply, +Options 39 http_delete/3, % +URL, -Reply, +Options 40 http_post/4, % +URL, +In, -Reply, +Options 41 http_put/4, % +URL, +In, -Reply, +Options 42 http_patch/4, % +URL, +In, -Reply, +Options 43 http_read_data/3, % +Header, -Data, :Options 44 http_disconnect/1 % +What 45 ]). 46:- autoload(http_header,[http_post_data/3]). 47:- autoload(http_stream,[http_chunked_open/3,stream_range_open/3]). 48:- autoload(library(error),[must_be/2]). 49:- autoload(library(lists),[delete/3,select/3]). 50:- autoload(library(memfile), 51 [ new_memory_file/1, open_memory_file/4, free_memory_file/1, 52 memory_file_to_atom/3, memory_file_to_string/3, 53 memory_file_to_codes/3, open_memory_file/3 54 ]). 55:- autoload(library(option), 56 [option/3,option/2,meta_options/3,select_option/3]). 57:- autoload(library(uri),[uri_query_components/2]). 58:- autoload(library(http/http_open), 59 [http_open/3,http_close_keep_alive/1]). 60 61:- meta_predicate 62 http_read_data(, , ). 63 64:- multifile 65 http_convert_data/4, % http_read_data plugin-hook 66 http:post_data_hook/3. 67 68:- predicate_options(http_get/3, 3, 69 [ pass_to(http_open:http_open/3, 3), 70 pass_to(http_open:http_read_data/3, 3), 71 pass_to(http_json:http_read_json/3, 3) 72 ]). 73:- predicate_options(http_delete/3, 3, [pass_to(http_get/3, 3)]). 74:- predicate_options(http_post/4, 4, [pass_to(http_get/3, 3)]). 75:- predicate_options(http_put/4, 4, [pass_to(http_post/4, 4)]). 76:- predicate_options(http_read_data/3, 3, 77 [ to(any), 78 content_type(any), 79 form_data(oneof([form,mime])), 80 input_encoding(encoding), % multipart messages 81 on_filename(callable), 82 module(atom), % x-prolog data 83 variable_names(-list) 84 ]). 85 86 87/** <module> HTTP client library 88 89This library provides the four basic HTTP client actions: =GET=, 90=DELETE=, =POST= and =PUT=. In addition, it provides http_read_data/3, 91which is used by library(http/http_parameters) to decode =POST= data in 92server applications. 93 94This library is based on http_open/3, which opens a URL as a Prolog 95stream. The reply is processed by http_read_data/3. The following 96content-types are supported. Options passed to http_get/3 and friends 97are passed to http_read_data/3, which in turn passes them to the 98conversion predicates. Support for additional content types can be added 99by extending the multifile predicate http_client:http_convert_data/4. 100 101 - 'application/x-www-form-urlencoded' 102 Built in. Converts form-data into a list of `Name=Value` terms. 103 - 'application/x-prolog' 104 Built in. Reads a single Prolog term. 105 - 'multipart/form-data' 106 Processed if library(http/http_multipart_plugin) is loaded. This 107 format should be used to handle web forms that upload a file. 108 - 'text/html' | 'text/xml' 109 Processed if library(http/http_sgml_plugin) is loaded. See load_html/3 110 for details and load_xml/3 for details. The output is often processed 111 using xpath/3. 112 - 'application/json' | 'application/jsonrequest' 113 Processed if library(http/http_json) is loaded. The option 114 json_object(As) can be used to return a term json(Attributes) 115 (`As` is `term`) or a dict (`As` is `dict`). 116*/ 117 118 /******************************* 119 * GET * 120 *******************************/ 121 122%! http_get(+URL, -Data, +Options) is det. 123% 124% Get data from a URL server and convert it to a suitable Prolog 125% representation based on the =|Content-Type|= header and plugins. 126% This predicate is the common implementation of the HTTP client 127% operations. The predicates http_delete/3, http_post/4 and 128% http_put/4 call this predicate with an appropriate 129% method(+Method) option and ---for http_post/4 and http_put/4--- 130% a post(+Data) option. 131% 132% Options are passed to http_open/3 and http_read_data/3. Other 133% options: 134% 135% - reply_header(-Fields) 136% Synonym for headers(Fields) from http_open/3. Provided for 137% backward compatibility. Note that http_version(Major-Minor) 138% is missing in the new version. 139 140http_get(URL, Data, Options) :- 141 headers_option(Options, Options1, Headers), 142 option(reply_header(Headers), Options, _), 143 http_open(URL, In, Options1), 144 delete(Headers, transfer_encoding(_), Headers1), 145 call_cleanup( 146 http_read_data(In, Headers1, Data, Options), 147 close(In)). 148 149headers_option(Options, Options1, Headers) :- 150 option(headers(Headers), Options), 151 !, 152 Options1 = Options. 153headers_option(Options, [headers(Headers)|Options], Headers). 154 155 156%! http_delete(+URL, -Data, +Options) is det. 157% 158% Execute a =DELETE= method on the server. Arguments are the same 159% as for http_get/3. Typically one should pass the option 160% status_code(-Code) to assess and evaluate the returned status 161% code. Without, codes other than 200 are interpreted as an error. 162% 163% @tbd Properly map the 201, 202 and 204 replies. 164% @see Implemented on top of http_get/3. 165 166http_delete(URL, Data, Options) :- 167 http_get(URL, Data, [method(delete)|Options]). 168 169 170%! http_post(+URL, +Data, -Reply, +Options) is det. 171% 172% Issue an HTTP =POST= request. Data is posted using 173% http_post_data/3. The HTTP server reply is returned in Reply, 174% using the same rules as for http_get/3. 175% 176% @see Implemented on top of http_get/3. 177 178http_post(URL, Data, Reply, Options) :- 179 http_get(URL, Reply, 180 [ post(Data) 181 | Options 182 ]). 183 184%! http_put(+URL, +Data, -Reply, +Options) 185% 186% Issue an HTTP =PUT= request. Arguments are the same as for 187% http_post/4. 188% 189% @see Implemented on top of http_post/4. 190 191http_put(URL, In, Out, Options) :- 192 http_post(URL, In, Out, [method(put)|Options]). 193 194%! http_patch(+URL, +Data, -Reply, +Options) 195% 196% Issue an HTTP =PATCH= request. Arguments are the same as for 197% http_post/4. 198% 199% @see Implemented on top of http_post/4. 200 201http_patch(URL, In, Out, Options) :- 202 http_post(URL, In, Out, [method(patch)|Options]). 203 204%! http_read_data(+Request, -Data, +Options) is det. 205% 206% Read data from an HTTP connection and convert it according to 207% the supplied to(Format) option or based on the =|Content-type|= 208% in the Request. The following options are supported: 209% 210% * to(Format) 211% Convert data into Format. Values are: 212% - stream(+WriteStream)) 213% Append the content of the message to Stream 214% - atom 215% Return the reply as an atom 216% - string 217% Return the reply as a string 218% - codes 219% Return the reply as a list of codes 220% * form_data(AsForm) 221% * input_encoding(+Encoding) 222% * on_filename(:CallBack) 223% These options are implemented by the plugin 224% library(http/http_multipart_plugin) and apply to processing 225% =|multipart/form-data|= content. 226% * content_type(+Type) 227% Overrule the content-type that is part of Request as a 228% work-around for wrongly configured servers. 229% 230% Without plugins, this predicate handles 231% 232% * 'application/x-www-form-urlencoded' 233% Converts form-data into a list of `Name=Value` terms. 234% * 'application/x-prolog' 235% Converts data into a Prolog term. 236% 237% @param Request is a parsed HTTP request as returned by 238% http_read_request/2 or available from the HTTP server's request 239% dispatcher. Request must contain a term input(In) that provides 240% the input stream from the HTTP server. 241 242http_read_data(Fields, Data, QOptions) :- 243 meta_options(is_meta, QOptions, Options), 244 memberchk(input(In), Fields), 245 ( catch(http_read_data(In, Fields, Data, Options), 246 error(Formal,_), 247 throw(error(Formal, context(_, in_http_request)))) 248 -> true 249 ; throw(error(failed(http_read_data), _)) 250 ). 251 252is_meta(on_filename). 253 254http_read_data(_In, Fields, Data, _Options) :- 255 option(status_code(Code), Fields), 256 no_content_status(Code), 257 \+ ( option(content_length(Len), Fields), 258 Len > 0 259 ), 260 !, 261 Data = ''. 262http_read_data(In, Fields, Data, Options) :- % Transfer-encoding: chunked 263 select(transfer_encoding(chunked), Fields, RestFields), 264 !, 265 setup_call_cleanup( 266 http_chunked_open(In, DataStream, []), 267 http_read_data(DataStream, RestFields, Data, Options), 268 close(DataStream)). 269http_read_data(In, Fields, Data, Options) :- 270 option(to(X), Options), 271 !, 272 ( X = stream(Stream) 273 -> ( memberchk(content_length(Bytes), Fields) 274 -> copy_stream_data(In, Stream, Bytes) 275 ; copy_stream_data(In, Stream) 276 ) 277 ; must_be(oneof([atom,string,codes]), X), 278 setup_call_cleanup( 279 new_memory_file(MemFile), 280 ( setup_call_cleanup( 281 open_memory_file(MemFile, write, Stream, 282 [encoding(octet)]), 283 ( memberchk(content_length(Bytes), Fields) 284 -> copy_stream_data(In, Stream, Bytes) 285 ; copy_stream_data(In, Stream) 286 ), 287 close(Stream)), 288 encoding(Fields, Encoding, Options), 289 memory_file_to(X, MemFile, Encoding, Data0) 290 ), 291 free_memory_file(MemFile)), 292 Data = Data0 293 ). 294http_read_data(In, Fields, Data, _) :- 295 option(content_type(ContentType), Fields), 296 is_content_type(ContentType, 'application/x-www-form-urlencoded'), 297 !, 298 http_read_data(In, Fields, Codes, [to(string)]), 299 uri_query_components(Codes, Data). 300http_read_data(In, Fields, Data, Options) :- % call hook 301 ( select_option(content_type(Type), Options, Options1) 302 -> delete(Fields, content_type(_), Fields1), 303 http_convert_data(In, [content_type(Type)|Fields1], Data, Options1) 304 ; http_convert_data(In, Fields, Data, Options) 305 ), 306 !. 307http_read_data(In, Fields, Data, Options) :- 308 http_read_data(In, Fields, Data, [to(atom)|Options]). 309 310memory_file_to(atom, MemFile, Encoding, Data) :- 311 memory_file_to_atom(MemFile, Data, Encoding). 312memory_file_to(string, MemFile, Encoding, Data) :- 313 memory_file_to_string(MemFile, Data, Encoding). 314memory_file_to(codes, MemFile, Encoding, Data) :- 315 memory_file_to_codes(MemFile, Data, Encoding). 316 317 318encoding(_Fields, Encoding, Options) :- 319 option(input_encoding(Encoding), Options), 320 !. 321encoding(Fields, utf8, _) :- 322 memberchk(content_type(Type), Fields), 323 ( sub_atom(Type, _, _, _, 'UTF-8') 324 -> true 325 ; sub_atom(Type, _, _, _, 'utf-8') 326 ), 327 !. 328encoding(_, octet, _). 329 330is_content_type(ContentType, Check) :- 331 sub_atom(ContentType, 0, Len, After, Check), 332 ( After == 0 333 -> true 334 ; sub_atom(ContentType, Len, 1, _, ';') 335 ). 336 337%! no_content_status(+Code) is semidet. 338% 339% True when Code is an HTTP status code that carries no content. 340% 341% @see Issue#157 342 343no_content_status(Code) :- 344 between(100, 199, Code), 345 !. 346no_content_status(204). 347 348%! http_convert_data(+In, +Fields, -Data, +Options) is semidet. 349% 350% Multi-file hook to convert a HTTP payload according to the 351% _Content-Type_ header. The default implementation deals with 352% application/x-prolog. The HTTP framework provides 353% implementations for JSON (library(http/http_json)), HTML/XML 354% (library(http/http_sgml_plugin)) 355 356http_convert_data(In, Fields, Data, Options) :- 357 memberchk(content_type(Type), Fields), 358 is_content_type(Type, 'application/x-prolog'), 359 !, 360 ( memberchk(content_length(Bytes), Fields) 361 -> setup_call_cleanup( 362 ( stream_range_open(In, Range, [size(Bytes)]), 363 set_stream(Range, encoding(utf8)), 364 set_stream(Range, file_name('HTTP:DATA')) 365 ), 366 read_term(Range, Data, Options), 367 close(Range)) 368 ; set_stream(In, encoding(utf8)), 369 read_term(In, Data, Options) 370 ). 371 372%! http_disconnect(+Connections) is det. 373% 374% Close down some connections. Currently Connections must have the 375% value =all=, closing all connections. 376% 377% @deprecated New code should use http_close_keep_alive/1 from 378% library(http/http_open). 379 380http_disconnect(all) :- 381 http_close_keep_alive(_). 382 383%! http:post_data_hook(+Term, +Out, +Options) is semidet. 384% 385% Hook to extend the datatypes supported by the post(Data) option 386% of http_open/3. The default implementation supports 387% prolog(Term), sending a Prolog term as =|application/x-prolog|=. 388 389httppost_data_hook(prolog(Term), Out, HdrExtra) :- 390 setup_call_cleanup( 391 ( new_memory_file(MemFile), 392 open_memory_file(MemFile, write, Handle) 393 ), 394 ( format(Handle, 395 'Content-Type: application/x-prolog; charset=UTF-8~n~n', 396 []), 397 write_term(Handle, Term, 398 [ quoted(true), 399 ignore_ops(true), 400 fullstop(true), 401 nl(true) 402 ]) 403 ), 404 close(Handle)), 405 setup_call_cleanup( 406 open_memory_file(MemFile, read, RdHandle, 407 [ free_on_close(true) 408 ]), 409 http_post_data(cgi_stream(RdHandle), Out, HdrExtra), 410 close(RdHandle))