In 2015, Australia passed a new piece of legislation entitled the Telecommunications (Interception and Access) Amendment (Data Retention) Act 2015 . Following the introduction of this act, service providers have obligations to retain various data associated with services provided to customers.

Despite having taken effect in October 2015, I still see a lot of confusion in the service provider and broader community about exactly what customer data should be and is retained by providers. Having been heavily involved at my work in both preparing our implementation plan and providing guidance to the service provider industry at large, I feel I’m somewhat cognizant of some of the common misunderstandings and hope I can shed some light on how providers should be interpreting their obligations.

Disclaimer: I am not a lawyer and this is not advice. If you think your organisation may have a metadata obligation - the best thing you can do is contact a lawyer who is familiar with the service provider industry to get expert advice. Likewise, this website and these words are not those of my employer, so please don’t hold them accountable for any opinions herein.

Finally, a lot of this information (and much more) can be found in the document Data Retention - Frequently Asked Questions for Industry published by the Attorney Generals Department (AGD). The AGD copped a lot of flak from members of industry for not being able to clearly articulate how to interpret the legislation, however I’ve found this document (even in its first revision) very capable of doing so - if only people take the time and effort to read an understand it with a sense of calm.

A bit of context

First thing first - this law is in effect now. If you have a metadata obligation you are assumed to be compliant from the 13th October 2015 unless you have lodged what is known as a Data Retention Implementation Plan (DRIP) with the Communications Access Coordinator (CAC) and received approval prior to this date.

Providers with approved implementation plans may have up to the 12th April 2017 (18 months later) to become compliant.

Why do this

Quite simply, this act mandates service providers to retain data about customers buying a relevant service and some metadata around the service.

This information is often requested by police investigating issues, but historically has never been retained by service providers. By introducing this legislation, the government helps support law enforcement by ensuring important information must be retained.

Criticisms

There has been a lot of criticism around this legislation, from my reading this falls broadly into one of two areas:

• What is being collected - Misinformation about the data being collected, or misunderstanding of what is being asked of providers by overzealous industry members; and

• Who can access the data - Misinformation about who can request access to the data from a provider.

The legislation provides a very specific (and in my opinion reasonable) list of government agencies that can request access to this data. The Attorney general can also declare additions to this, however such appointments are public and thus have a level of oversight.

I have heard reports of agencies outside this list making requests for information from service providers and whilst it is unclear whether the providers have made data available when they shouldn’t have, it is clear that there is still confusion about who can ask what. The CAC is supposedly able to provide clarification on such matters to those that find themselves in this situation.

Exemptions

Providers are also able to apply for exemption to their obligations, both as part of an implementation plan and on an ongoing basis. An application for exemption may be made to the CAC on the basis of one of the following:

• the long-term interests of end-users of carriage services or of services provided by means of carriage services; or
• the efficiency and international competitiveness of the Australian telecommunications industry; or
• the availability of accessible and affordable carriage services that enhance the welfare of Australians

Exemptions are required to be kept confidential by providers simply because public knowledge of these loopholes may provide a vector for bad actors to exploit.

Funding

The Australian government has made funding available to industry in the form of the Data Retention Industry Grants Programme to implement their metadata obligations.

Broadly speaking, applications for funding were open to providers who incurred costs preparing their DRIP; performed work to ensure compliance between 30 October 2014 and 13 October 2015; or had an implementation plan approved to become compliant.

Grants totalling up to $128.4 million were awarded to applicants in August 2016 and information on the recipients and the allocation methodology are avilable on the AGD website.

There has been some level of controversy within industry regarding the grants awarded - some spectators have questioned funding requests disproportionate to the size of the providers operations.

It’s worth bearing in mind however, that as with most grant programmes - recipients must agree to a funding agreement which includes reporting requirements on how the money is being spent.

What data

Phew, okay - now we’re ready to talk about what information is being collected…

Information is only required to be collected where the customer has a service where a metadata obligation exists (evluating this is covered in the next section)

Generally speaking, if the provider does not handle or generate any of the data covered here - they are not required to generate it or capture it solely for the purpose of data retention.

Any information that is covered and available must be retained by the service provider for no less than two years.

Providers are required to ensure that data retained is:

• Adequately protected as required under the Privacy Act 1988
• Encrypting data at rest and providing access measures

Customer information

A provider must retain any customer contact (name, address, etc) or billing details in their CRM including historic data of at least 2 years.

Communication Metadata

Where the service provider facilitates a communication, the relevant metadata must be collected where available for any communication held or attempted to be held:

• The date and time (and where relevant, the duration) of a communication
• The source of a communication (who initiated the communication)
• The destination of a communication (who was being communicated to)
• The type of communication
• The location of the communication at the start and end of the communication (where relevant)

In the case the provider is able to positively identify the other party of a communication (ie: the other party is also a customer of the provider) then metadata about that customer must also be retained. It is my understanding that even if that other party does not consume any services that would be subject to metadata retention, though I have not sought any clarification on this).

It’s really important to note here that:

• The above items need to be considered in the context of the applicable service and may not apply
• Providers do not need to generate or collect metadata for third-party services that a customer consumes “over the top” of a service (with the exception of some specific services)

What’s NOT collected

Anything that falls outside of the items discussed above.

Specifically, it’s worth noting some specific things that aren’t covered:

• Content of communications (eg: email content, voip calls, etc)
• Web Browsing History - this is explicitly exempted by the legislation
• Customer location when service not in use (eg: mobile phone location)
• Packet metadata (eg: NetFlow, sFlow)

The last item here is one of the things overzealous operators have jumped on - however the AGD provides specific guidance around this.

Evaluating your obligation

Metadata retention obligations are determined on a per-service basis and a provider must consider each of the following criteria.

If the provider:service combination does not meet all the criteria below, there is no obligation.

Note: Again I would like to acknowledge the fantastic Industry FAQ from the AGD from which this is derived.

Am I a Carrier, CSP or ISP?

Are you one of the following:

• Carrier
• Internet Service Provider (ISP)
• Carriage Service Provider (CSP)

These are very well defined terms under existing Telecommunications legislation and generally you will know if you are one of these.

It’s worth noting that if you provide certain types of listed carriage services to a third party in return for a reward, you are considered a CSP.

Is it a relevant service

Does the service carry, or enable a communication to be carried out?

This doesn’t include services required to carry out a communication (eg: DNS), just those that actually carry it.

Intent is important here, if the service isn’t primarily concerned with carrying communications in normal operation - you don’t need to anticipate off-the-wall scenarios (eg: iodine dns, etc).

Not in “same area”

Services offered and consumed within the same property boundary are exempted.

Not in “immediate circle”

Metadata obligations do not extend to services offered to officers or employees of the provider.

Worked examples

Here are some pretty common scenarios that come up and how I would evaluate a metadata obligation for them.

Wired internet connection

There is an obligation unless the “immediate circle” exclusion applies.

Customer data would be readily available. IP Address allocation and session start/stop times are acceptable metadata records. The service address of the service would suffice for the this type of connection.

Telephony Provider

I’ve added this example mostly to cover off voice services, whilst they remain a large part of the focus of this legislation - the industry is very mature and has a solid background in generating and retaining metadata here.

Generally speaking, there’s an obligation here except free services.

CRM, telephone number allocation, attempted inbound/outbound call logs are required here. Including physical location of the handset (fixed address or mobile) at the call initiation/hangup.

Free Public Wi-Fi

If the operator of the WiFi is not a Carrier/ISP there is no obligation. This is because the service is offered for free so they would not be considered a CSP.

If the operator of the WiFi is a Carrier/ISP/CSP but the service is offered within a single property boundary there is no obligation. This is because the “same area” exclusion applies.

If the operator of the WiFi is a Carrier/ISP/CSP and the service is offered across multiple locations… get a lawyer. Strictly speaking you don’t meet the “same area” exclusion - but some good lawyering might just change that.

Where an obligation exists, depending on the solution you may not have customer identifying information. MAC addresses suffice if your captive portal collects them - however not all solutions do.

If you are performing NAT you may be required to collect NAT mappings.

Paid Hotel WiFi

The hotel may considered a CSP as they are selling internet access for reward. Fortunately for them, the “same area” exclusion may apply to these providers.

Unless they’re a chain of hotels, in which case clever lawyering may be required.

Or if they outsource the operation of the WiFi, in which case the operator almost certainly has an obligation.

Staff email accounts

There is no obligation - this is because of the “immediate circle” exclusion as email accounts are offered to employees only.

Unless email is a proscribed service for CSP (I don’t think it is, but I haven’t looked), and you outsource your IT externally - then your IT provider shouldn’t have an obligation either unless they’re considered a CSP/Carrier/ISP for other business activities.

If they were, your supply agreement may determine whether they have an obligation - if they’re contracted to perform professional services for your staff email server - there is no obligation, however if they are providing the email as a contracted service then that might be a different story.

Even then, the “same area” restriction may apply if they manage a server on your premises.

Shared web hosting

There is no metadata obligation.

Whilst you operate this service, web browsing history is specifically excluded form legislation - so web server logs are not in scope here.

Any over-the top services operated by your customer (forums, etc) are not your responsibility to retain data for.

However, in theory any outbound traffic generated by a customer may be in scope - in which case the data you retain may need to include process owner (if you give customers a system account and run their apps as them), as this is a similar situation as NAT (shared resource, retain the mappings).

If I was a hosting provider, I’d be getting a lawyer to review this with me. I’d also be preparing an application for exemption on the basis that this data would not normally be generated as part of business as usual operation, even if you offer a dedicated IP for SSL/other reasons.

VPS hosting

There is a metadata obligation here.

In the case of VPS, a static IP allocation exists and meeting the obligation is quite easy to meet.

Conclusion

I hope this helps address some of the landscape of metadata retention in Australia.

The general constraints are fairly easy to understand for engineering staff looking after these services, however your obligation does depend on how you offer the service from both a technical and commercial point of view.

My only advice is that you find (and retain) a competent technology lawyer and keep a level head!