util¶

url builder related utility methods.

crawlib.util.add_params(endpoint, params)¶

Combine query endpoint and params.

Example:

>>> add_params("https://www.google.com/search", {"q": "iphone"})
https://www.google.com/search?q=iphone

crawlib.util.get_domain(url)¶

Get domain part of an url.

crawlib.util.get_netloc(url)¶

Get network location part of an url.

For example: https://www.python.org/doc/ -> www.python.org

crawlib.util.join_all(domain, *parts)¶

Join all url components.

Example:

>>> join_all("https://www.apple.com", "iphone")
https://www.apple.com/iphone

Parameters:	domain – Domain parts, example: https://www.python.org parts – Other parts, example: “/doc”, “/py27”
Returns:	url

crawlib