New in 8.0.6: system.net.httpClient()

Coming in 8.0.6, and available in nightly builds as of today, a not-as-pretty but hopefully pretty exciting new feature: system.net.httpClient. This is a new function that will deprecate the existing system.net.http* functions. I’m excited for users to try it out. It wraps the new-to-Java-11 HttpClient with some convenience stuff for usage from Python. Full details are below, but basic usage is pretty similar to the existing functions, just much more powerful:

client = system.net.httpClient()
promise = client.getAsync("https://httpbin.org/get", params={"a": 1, "b": 2})
print "doing something while waiting"
response = promise.get()
if response.good:
    print response.json['args']['a']

A key feature are asynchronous requests - each common HTTP verb (get, post, put, patch, etc) is covered with a matching asynchronous variant, plus there are generic fallback request()/requestAsync() methods to enable custom HTTP verbs to be used.

Usage Examples

Synchronous (Blocking)

client = system.net.httpClient()
print client
>>> <JythonHttpClient@1072310501 timeout=10000, bypass_cert_validation=False, has_credentials=False, proxy=None, cookie_policy='ACCEPT_ORIGINAL_SERVER', redirect_policy='NEVER'>

Create a new JythonHttpClient instance with default parameters. This client accepts most common HTTP methods (GET, POST, PATCH, etc) directly, as well as arbitrary HTTP methods (.request())

response = client.get("https://httpbin.org/get")
print response
>>> <Response@1848533253 'https://httpbin.org/get' [200]>

Run a simple GET request against a URL. Returns a Response object, which has various convenience methods, such as .text, .json, etc.

print response.json
>>> {'headers': {'User-Agent': u'Ignition', 'Host': u'httpbin.org'}, 'origin': u'70.96.196.62, 70.96.196.62', 'url': u'https://httpbin.org/get', 'args': {}}

Return the JSON body of a response directly as a Python object (most likely, a dictionary containing other dictionaries). If the body can’t be decoded as JSON, throws an error.

Complete Example

client = system.net.httpClient()
response = client.get("https://httpbin.org/get", params={"a": 1, "b": 2})
if response.good:
    print response.json['args']['a']

Asynchronous (Non-Blocking)

def callback(arg):
    ## do something out of band
    print 'asyncHandler', arg

First, define a callback function, which will be executed with the result of the HTTP request (or the error, if an error was thrown during evaluation). All httpClient functions return a Promise that contains a Response object.

c = system.net.httpClient()
p = c.getAsync("https://httpbin.org/status/404")

Then, create your client and call the async variant of an HTTP method. This will directly return a Promise object.

p.whenComplete(callback)

Add your callback to the Promise object. The whenComplete(callback) handler will fire whenever the Promise resolves, but is not expected to return anything, in contrast with .then(callback), which is expected to return a new value which will itself be wrapped in a Promise, allowing for chaining.

print p
print p.get()
>>> <Promise@1427747612 isDone=false>
<Response@1299364187 'https://httpbin.org/status/404' [404]>

Finally, you can also directly return the result by calling .get([timeout]). If not specified, the timeout defaults to 60 seconds. This is optional - you can fire off asynchronous requests and they will be automatically executed, even if you don’t ever ask them to block.

Complete Example

client = system.net.httpClient()
promise = client.getAsync("https://httpbin.org/get", params={"a": 1, "b": 2})
print "doing something while waiting"
response = promise.get()
if response.good:
    print response.json['args']['a']

system.net.httpClient()

Creates a new JythonHttpClient instance.

Parameters

Parameter Type Default Value Description
timeout int 60000 A value, in milliseconds, to set the client’s connect timeout setting to.
bypass_cert_validation bool False A boolean indicating whether to bypass SSL verification on the target host. Pass True to skip verification.
username str None A string indicating the username to use for authentication if the remote server requests authentication; specifically, by responding with a WWW-Authenticate or Proxy-Authenticate header. Only supports Basic authorization. If username is present but not password, an empty string will be used for password.
password str None A string indicating the password to use for authentication.
proxy str None The address of a proxy, which will be used for HTTP and HTTPS traffic. If the port is not specified, it will be assumed from the protocol in the URL, i.e. 80/443.
cookie_policy str “ACCEPT_ORIGINAL_SERVER” A string representing this client’s cookie policy. Accepts values ACCEPT_ALL, ACCEPT_NONE, and ACCEPT_ORIGINAL_SERVER.
redirect_policy str “NORMAL” A string representing this client’s redirect policy. Accepts values NEVER, ALWAYS, and NORMAL (allows redirects, except HTTPS -> HTTP downgrades.
customizer Callable None A reference to a callable function. This function will be called with one argument (an instance of HttpClient.Builder). The function should operate on the builder class and allows for arbitrary customization of the created HTTP client.

JythonHttpClient

A wrapper around Java’s HttpClient.

Methods

.get(), .post(), .put(), .delete(), .head(), .patch(), .options(), .trace() -> Response

Dispatch an HTTP request. Returns a Response object.

Parameters
Parameter Type Default Value Description
url str Required The URL to connect to.
parameters str, dict None Parameters to append to the URL. If supplied as a string, will be directly appended to the URL. If supplied as a dictionary, key/value pairs will be automatically URL encoded.
data* str None Data to send in the request. String data will be sent with a Content-Type of text/plain; charset=UTF-8, unless the Content-Type header was already specified.
data* dict None Data to send in the request. Will be automatically encoded into JSON, and set the Content-Type header to application/json;charset=UTF-8 unless the Content-Type header was already specified.
data* byte[] None Data to send in the request. A byte array will be sent directly, and set the Content-Type header to application/octet-stream unless the Content-Type header was already specified.
file str None The path to a file relative to the HTTP client. If specified, and the path is valid, will directly send the data in the file. The file attribute overrides any value set in data; only the file’s data will be sent.
headers dict None A dictionary of headers to add to the request.
username str None Username to add to a Basic Authorization header in the outgoing request. If username is specified, but not password, password is assumed to be an empty string.
password str None Password to add to a Basic Authorization header in the outgoing request.
timeout int 60000 Timeout for this request, in milliseconds.

.request() -> Response

Same as the above functions, but with an additional required argument, method:

Parameters
Parameter Type Default Value Description
url str Required The URL to connect to.
method str Required The method to use in the request.
parameters str, dict None Parameters to append to the URL. If supplied as a string, will be directly appended to the URL. If supplied as a dictionary, key/value pairs will be automatically URL encoded.
data* str None Data to send in the request. String data will be sent with a Content-Type of text/plain; charset=UTF-8, unless the Content-Type header was already specified.
data* dict None Data to send in the request. Will be automatically encoded into JSON, and set the Content-Type header to application/json;charset=UTF-8 unless the Content-Type header was already specified.
data* byte[] None Data to send in the request. A byte array will be sent directly, and set the Content-Type header to application/octet-stream unless the Content-Type header was already specified.
file str None The path to a file relative to the HTTP client. If specified, and the path is valid, will directly send the data in the file. The file attribute overrides any value set in data; only the file’s data will be sent.
headers dict None A dictionary of headers to add to the request.
username str None Username to add to a Basic Authorization header in the outgoing request. If username is specified, but not password, password is assumed to be an empty string.
password str None Password to add to a Basic Authorization header in the outgoing request.
timeout int 60000 Timeout for this request, in milliseconds.

.getAsync(), .postAsync(), .putAsync(), .deleteAsync(), .headAsync(), .patchAsync(), .optionsAsync(), .traceAsync() -> Promise

Returns a new Promise instance.

Parameters
Parameter Type Default Value Description
url str Required The URL to connect to.
parameters str, dict None Parameters to append to the URL. If supplied as a string, will be directly appended to the URL. If supplied as a dictionary, key/value pairs will be automatically URL encoded.
data* str None Data to send in the request. String data will be sent with a Content-Type of text/plain; charset=UTF-8, unless the Content-Type header was already specified.
data* dict None Data to send in the request. Will be automatically encoded into JSON, and set the Content-Type header to application/json;charset=UTF-8 unless the Content-Type header was already specified.
data* byte[] None Data to send in the request. A byte array will be sent directly, and set the Content-Type header to application/octet-stream unless the Content-Type header was already specified.
file str None The path to a file relative to the HTTP client. If specified, and the path is valid, will directly send the data in the file. The file attribute overrides any value set in data; only the file’s data will be sent.
headers dict None A dictionary of headers to add to the request.
username str None Username to add to a Basic Authorization header in the outgoing request. If username is specified, but not password, password is assumed to be an empty string.
password str None Password to add to a Basic Authorization header in the outgoing request.
timeout int 60000 Timeout for this request, in milliseconds.

.requestAsync() -> Promise

Same as the above functions, but with an additional required argument, method:

Parameters
Parameter Type Default Value Description
url str Required The URL to connect to.
method str Required The method to use in the request.
parameters str, dict None Parameters to append to the URL. If supplied as a string, will be directly appended to the URL. If supplied as a dictionary, key/value pairs will be automatically URL encoded.
data* str None Data to send in the request. String data will be sent with a Content-Type of text/plain; charset=UTF-8, unless the Content-Type header was already specified.
data* dict None Data to send in the request. Will be automatically encoded into JSON, and set the Content-Type header to application/json;charset=UTF-8 unless the Content-Type header was already specified.
data* byte[] None Data to send in the request. A byte array will be sent directly, and set the Content-Type header to application/octet-stream unless the Content-Type header was already specified.
file str None The path to a file relative to the HTTP client. If specified, and the path is valid, will directly send the data in the file. The file attribute overrides any value set in data; only the file’s data will be sent.
headers dict None A dictionary of headers to add to the request.
username str None Username to add to a Basic Authorization header in the outgoing request. If username is specified, but not password, password is assumed to be an empty string.
password str None Password to add to a Basic Authorization header in the outgoing request.
timeout int 60000 Timeout for this request, in milliseconds.

Attributes

.javaClient/.getJavaClient() -> HttpClient

Returns the underlying Java HTTPClient.

.cookieManager/.getCookieManager()

Returns a CookieManager, which can be used to override the cookie storage policy (setCookiePolicy(policy), where policy is a java.net.CookiePolicy), or to retrieve the CookieStore directly via (getCookieStore()). The CookieStore will itself have various methods to add or retrieve cookies, as detailed on the CookieStore interface.

Response

A wrapper around a Java HttpResponse.

Methods

.body/.getBody() -> byte[]

Returns the response content directly.

.json/.getJson([encoding]) -> dict

Return the response content, with the encoding specified by the response (or the encoding specified by the encoding argument, decoded into Python (dictionary, list, etc). If response can’t be decoded to JSON, throws an error.

.text/.getText([encoding]) -> str

Return the response content decoded into a string - either with the charset specified in the response (or UTF-8 if not specified), or using the encoding specified in the function call.

Attributes

.statusCode/.getStatusCode() -> int

Return the status code of the response object, i.e. 200 or 404.

.good/.isGood() -> bool

Returns True if the response was good (i.e. 200) or False if it was a client or server error (status code between 400 and 599).

.clientError/.isClientError() -> bool

Returns True if the response was a client error, as in an HTTP 4XX response.

.serverError/.isServerError() -> bool

Returns True if the response was a server error, as in an HTTP 5XX response.

.url/.getUrl() -> str

Returns the URL this Response connected to.

.headers/.getHeaders() -> dict

Returns a case-insensitive “dictionary” of headers present on the response. Values will be in a list, even if only a single value is present.

.javaResponse/.getJavaResponse() -> HttpResponse

Returns the underlying Java HttpResponse behind this Response.

.cookieManager/.getCookieManager() -> CookieManager

Returns a CookieManager, which can be used to override the cookie storage policy (setCookiePolicy(policy), where policy is a java.net.CookiePolicy), or to retrieve the CookieStore directly via (getCookieStore()). The CookieStore will itself have various methods to add or retrieve cookies, as detailed on the CookieStore interface.

.request/.getRequest() -> RequestWrapper

Returns a RequestWrapper object, that thinly wraps a java.net.http.HttpRequest.

Promise

A wrapper around a Java CompletableFuture that returns some type

Methods

.get([timeout]) -> <T>

Block for timeout until a result is available. The result object can technically be any type, if chaining, but will be a Response object if calling one of the HTTPClient methods directly. If the timeout is met without a result, throws an exception. The default timeout is 60s, just like existing system.net.http* functions.

.then(callback) -> Promise

Allows chaining; returns a new Promise, wrapping callback. Callback should either accept two arguments (result, error) or have a single argument but be able to accept exceptions as well as valid values.

.handleException(callback) -> Promise

In the event of an exception in a potential chain of promises, handleException will be called with one argument (the thrown error) and is expected to return a new fallback value for the next step in the promise chain.

.whenComplete(callback) -> void

Call the provided callback when this promise finishes evaluating. Callback will be called with return value as the first argument, and any thrown error as the second argument. Any return value will be ignored.

.cancel() -> bool

Attempt to cancel the wrapped Java future. Returns True if the cancellation succeeded.

Attributes

.future/.getFuture()

Returns the underlying Java CompletableFuture object that this Promise contains.

.done/.isDone()

Returns True if the underlying future has completed - with a good result, or an exception.

RequestWrapper

A wrapper around a base Java HttpRequest.

Attributes

.url/.getUrl() -> str

Returns the actual URL that was contacted in the request.

.method/.getMethod() -> str

Return the HTTP method used in this request; GET, POST, PATCH, etc.

.headers/.getHeaders() -> dict

Returns a case-insensitive “dictionary” of headers present on the request. Values will always be in a list, even if only a single value is present.

.timeout/.getTimeout() -> int

Returns the timeout this query was set to, or -1 if timeout was invalid.

.version/.getVersion() -> str

Returns the HTTP version used for this request; either HTTP_1_1 or HTTP_2

.javaRequest/.getJavaRequest() -> HttpRequest

Returns the underlying HttpRequest object directly.

17 Likes

Now this is exciting. Was the existing system.net features written in FPMI? Looks like a 10+ year jump!

Close to, yeah. Oldest commit in our git history is 2011, and it has the original implementation of the functions, which mostly hasn’t changed since then.