API Documentation
Overview
Driftnet's Internet Scan data contains reports gathered by visiting open internet services using their IP and port.
Searching by IP
Internet scan data is returned by the scan/protocols
endpoint. The simplest way to search internet scan data is by IP address.
curl -s -H 'Authorization: Bearer <your-api-token>' \ 'https://api.driftnet.io/v1/scan/protocols?ip=8.8.8.8' \ | jq . \ | less -S
{ "page": 0, "pages": 22, "result_count": 2174, "results": [ { "date": "2019-05-13", "id": "hJgzWy25TfuNfhvgMzs8Tw", "items": [ { "context": "", "is_metadata": true, "type": "ip", "value": "8.8.8.8" }, { "context": "", "is_metadata": true, "type": "port-tcp", "value": "443" }, ...
In this example, there are 2174 results for IP address 8.8.8.8
. The API returns results in batches of 100, so there are 22 pages in total. The newest results are returned first. To retrieve another page of results, use the page=
parameter. Page numbering starts at zero.
Each result takes the form of a report. A report has a date stamp, a unique ID, and a collection of items.
Each item contains:
type
: The type of data being displayed, e.g.ip
orhost
.context
: The context in which the value was seen, e.g.cert-dns-name
for an ip or host seen inside an X.509 certificate.value
: The actual data value.is_metadata
:true
if the data came from inside the collection system (e.g. an enrichment),false
if it was collected from the external environment.
The ip
parameter can take a CIDR range. Setting ip=8.8.8.0/24
would have searched the entire /24 address range, i.e. all addresses from 8.8.8.0 to 8.8.8.255 inclusive.
Including indirect IPs
By default, a search by IP will only match cases where the IP being searched is the one that was scanned. You might also want to look for a type of ip
in any context, i.e. including results with some other reference (usually an X.509 certificate) to the IP address searched for. To get these results, add the indirect=true
qualifier:
curl -s -H 'Authorization: Bearer <your-api-token>' \ 'https://api.driftnet.io/v1/scan/protocols?ip=8.8.8.8&indirect=true' \ | jq . \ | less -S
{ "page": 0, "pages": 90, "result_count": 8922, "results": [ { "date": "2019-05-13", "id": "AR1NAVUQSMuy7NMQweNPgw", "items": [ { "context": "", "is_metadata": true, "type": "ip", "value": "45.160.122.135" }, { "context": "", "is_metadata": true, "type": "port-tcp", "value": "853" }, ...
With the indirect=true
parameter set we include results which present a certificate that is valid for IP 8.8.8.8, but which are not directly hosted on IP 8.8.8.8.
Expression searches
Driftnet supports much more complicated searches. For this, you can use expression=
queries.
Fundamentals
The basis of an expression search is a type and a value separated by a colon:
<type>:<value>
For example, to find all results with a server-banner
containing cherrypy
, you can call
curl -s -H 'Authorization: Bearer <your-api-token>' \ --get --data-urlencode 'expression=server-banner:cherrypy' \ 'https://api.driftnet.io/v1/scan/protocols' \ | jq . \ | less -S
You can search for almost any type field that you discover in the UI. Some other commonly-searched type fields include:
http-header
: To search within returned HTTP headers.host
: To search for a hostname, seen anywhere. Hostname matches are right-anchored, soexpression=host:example.com
will matchfoo.bar.example.com
, etc. As a special case, to search for a host field within a URL, also sethost_in_url=true
.title
: To search for the HTML title from a surveyed page.issuer
andsubject
: TLS certificate issuer/subject fields.url
: To search for a URL, seen anywhere. URL searches also support host queries, soexpression=url:example.com
will matchhttps://foo.bar.example.com/abc/def
, etc.
To get a degree of sloppy matching, use the slop=
parameter. For instance, setting slop=1
would allow a query for university london
to match university of london
.
If you don't know which type field to look in, you can just omit it entirely:
curl -s -H 'Authorization: Bearer <your-api-token>' \ --get --data-urlencode 'expression=cherrypy' \ 'https://api.driftnet.io/v1/scan/protocols' \ | jq . \ | less -S
Omitting the type field is slow compared to searching with a type, so try not to use this approach routinely.
Quoting Values
You should use double quotes when your value contains multiple words / spaces for instance to search for the entity
Tencent Cloud Computing you would use expression=entity:"Tencent Cloud Computing"
,
curl -s -H 'Authorization: Bearer <your-api-token>' \ --get --data-urlencode 'expression=entity:"Tencent Cloud Computing"' \ 'https://api.driftnet.io/v1/scan/protocols' \ | jq . \ | less -S
Combining Terms
You can combine <type>:<value>
terms using the operators and, or and not. These operators precede a <type>:<value>
term as shown in the following examples:
- To search a host AND port together:
host:driftnet.io and port-tcp:443
- To search for one product tag OR another:
product-tag:dahua or product-tag:hikvision
- To search for an entity but exclude two ports:
entity:"Tencent Cloud Computing" not port-tcp:22 and not port-tcp:2222
curl -s -H 'Authorization: Bearer <your-api-token>' \ --get --data-urlencode 'expression=entity:"Tencent Cloud Computing" not port-tcp:22 and not port-tcp:2222' \ 'https://api.driftnet.io/v1/scan/protocols' \ | jq . \ | less -S
Value Modifier Operators
For textual fields, <type>:<value>
terms will match words inside the text. In some situations you will want to have more control over how the value is interpreted, and there are two operators you can use in these situations: asterisk (*) for prefix searching and equals (=) for keyword searching.
Ending a value in an asterisk will use the value as a prefix. Replacing the colon between the type and value with an equals will only return results where the full value matches exactly. For example:
- To search for hikvision as a prefix inside the
product-tag
field:product-tag:hikvision*
- To search for an exact complete
product-tag
:product-tag="hikvision dvr ds-7208hwi-sh"
curl -s -H 'Authorization: Bearer <your-api-token>' \ --get --data-urlencode 'expression=product-tag="hikvision dvr ds-7208hwi-sh"' \ 'https://api.driftnet.io/v1/scan/protocols' \ | jq . \ | less -S
Special Keywords
There are two special keyword “types” in the API. Firstly,protocol
, which can be used to search for the context of the server-banner
type. This context actually contains the particular protocol that Driftnet identified.
curl -s -H 'Authorization: Bearer <your-api-token>' \ --get --data-urlencode 'expression=protocol=modbus' \ 'https://api.driftnet.io/v1/scan/protocols' \ | jq . \ | less -S
Secondly, http-header-name
can be used to search for the context of the http-header
type, i.e. for a particular HTTP header name.
curl -s -H 'Authorization: Bearer <your-api-token>' \ --get --data-urlencode 'expression=http-header-name=X-Clacks-Overhead' \ 'https://api.driftnet.io/v1/scan/protocols' \ | jq . \ | less -S
Both of these fields are case-sensitive, and both must be searched as full keywords (i.e. using the = operator).
Some types must also be searched as keywords. For instance, geo-country
type:
curl -s -H 'Authorization: Bearer <your-api-token>' \ --get --data-urlencode 'expression=geo-country=GB' \ 'https://api.driftnet.io/v1/scan/protocols' \ | jq . \ | less -S
Operator Precedence
The operator precedence is not > and > or. For example,
product-tag:dahua and geo-country=RU or geo-country=IR
is equal to
(product-tag:dahua and geo-country=RU) or geo-country=IR
To make sure you run the query you intended, it is a good idea to always use parentheses. The intent of the above query is properly captured by:
product-tag:dahua and (geo-country=RU or geo-country=IR)
curl -s -H 'Authorization: Bearer <your-api-token>' \ --get --data-urlencode 'expression=product-tag:dahua and (geo-country=RU or geo-country=IR)' \ 'https://api.driftnet.io/v1/scan/protocols' \ | jq . \ | less -S
Special Characters
Escape double quotes within a quoted string by using a backslash:
http-header:"inline; filename=\"index.html\""
curl -s -H 'Authorization: Bearer <your-api-token>' \ --get --data-urlencode 'expression=http-header="inline; filename=\"index.html\""' \ 'https://api.driftnet.io/v1/scan/protocols' \ | jq . \ | less -S
When searching without a type, you must also backslash-escape these characters:
; : = " * ( )
Context Restriction
It is possible to restrict searches to both a type and a context. Use a semicolon to specify the context, and specify a full keyword:
<type>;<context>=<value>
For example, to search a http-header
type with a specific context Server
and a particular value,
curl -s -H 'Authorization: Bearer <your-api-token>' \ --get --data-urlencode 'expression=http-header;Server="NimPlant C2 Server"' \ 'https://api.driftnet.io/v1/scan/protocols' \ | jq . \ | less -S
Usually, you"ll need to use the = (keyword) operator when including the context. The exceptions are for IP and host. For ip
searches, you should always use the : operator — e.g. to look up the IP address scanned,
curl -s -H 'Authorization: Bearer <your-api-token>' \ --get --data-urlencode 'expression=ip;:8.8.8.8' \ 'https://api.driftnet.io/v1/scan/protocols' \ | jq . \ | less -S
For host
searches you should use : if you want to include subdomains (expression=host;:example.com
), or = if you don't (expression=host;=example.com
).
Filtering
To time-filter the results, use the from=
and to=
parameters. These accept dates in the format YYYY-MM-DD
.
To filter on any arbitrary type, you may filter=type:value
, for instance filter=port-tcp:443
to restrict to TCP port 443. As an alternative to type filtering, you might just include want to include additional terms in your expression
.
Most-recent results
Driftnet stores results as a time series. Often, you only want to know the most recent result for an {ip, port}
pair. Set most_recent=true
, and voilà.
curl -s -H 'Authorization: Bearer <your-api-token>' \ --get --data-urlencode 'expression=ip;:8.8.8.8' --data-urlencode 'most_recent=true' \ 'https://api.driftnet.io/v1/scan/protocols' \ | jq . \ | less -S
Hosts in URLs
If you search for example.com
, do you also want to match URLs of the form scheme://sub.example.com/path/to/somewhere
? It depends on your use-case. By default, this feature is off, but you can enable it by setting host_in_url=true
:
curl -s -H 'Authorization: Bearer <your-api-token>' \ --get --data-urlencode 'expression=host:google.com' --data-urlencode 'host_in_url=true' \ 'https://api.driftnet.io/v1/scan/domains' \ | jq . \ | less -S
Summarization
Often, you want to get a quick rollup summary of a particular field. The API enables this with the summarize=
parameter. This call will get all scan results including TLS certificates issued to the University of Oxford, and summarize the ports they were seen on:
curl -s -H 'Authorization: Bearer <your-api-token>' \ --get --data-urlencode 'expression=subject:"university oxford"' --data-urlencode 'slop=1' \ --data-urlencode 'summarize=port-tcp' \ 'https://api.driftnet.io/v1/scan/protocols' \ | jq . \ | less -S
{ "summary": { "other": 0, "values": { "1000": 10, "10000": 11, "10443": 64, ... "943": 21, "9443": 9, "993": 10 } } }
The values
object in the return contains the extracted values, together with their counts.
You might want to restrict the summary to particular contexts, or to exclude particular contexts (e.g. to exclude summarize HTTP headers, or to exclude a particular HTTP header). You can use the summary_context=
and summary_nocontext=
parameters for this.
By default, the summary is limited to a maximum of 100 values; if there are more unique values than this, then the total count of non-summarized values is placed in other
. You can increase the maximum number of values in the summary using the summary_limit=
parameter, up to a ceiling of 10,000 values per call.
Enterprise users can increase the limit on the number of returned values up to a maximum of 1,000,000. For each block of 10,000 results returned, one unit of API quota will be consumed.
Enterprise users can include an overall cardinality count in the response by setting summary_cardinality=true
.
Prioritization
You can request Driftnet to schedule protocol-level collection on a particular IP/port pair by using the scan/protocols/prioritize
endpoint.
curl -s -H 'Authorization: Bearer <your-api-token>' \ 'https://api.driftnet.io/v1/scan/protocols/prioritize?ip=8.8.8.8&port=443' \ | jq .