Security can be an intimidating topic for web developers. The vocabulary is rich and full of acronyms. Trends evolve quickly as hackers and analysts play a perpetual cat-and-mouse game. Vulnerabilities stem from little details we cannot afford to spend too much time on during our day-to-day operations.

JavaScript developers already have a lot to take with the emergence of a new wave of innovative architectures, such as React Server Components, Next.js App Router, or Astro islands.

So, let’s have a focused approach. What we need is to be able to detect and palliate the most common security issues. A top ten of the most common vulnerabilities would be ideal.

Meet The OWASP Top 10

Guess what: there happens to be such a top ten of the most common vulnerabilities, curated by experts in the field!

It is provided by the OWASP Foundation, and it’s an extremely valuable resource for getting started with security.

OWASP stands for “Open Worldwide Application Security Project.” It’s a nonprofit foundation whose goal is to make software more secure globally. It supports many open-source projects and produces high-quality education resources, including the OWASP top 10 vulnerabilities list.

We will dive through each item of the OWASP top 10 to understand how to recognize these vulnerabilities in a full-stack application.

Note: I will use Next.js as an example, but this knowledge applies to any similar full-stack architecture, even outside of the JavaScript ecosystem.

Let’s start our countdown towards a safer web!

Number 10: Server-Side Request Forgery (SSRF)

You may have heard about Server-Side Rendering, aka SSR. Well, you can consider SSRF to be its evil twin acronym.

Server-Side Request Forgery can be summed up as letting an attacker fire requests using your backend server. Besides hosting costs that may rise up, the main problem is that the attacker will benefit from your server’s level of accreditation. In a complex architecture, this means being able to target your internal private services using your own corrupted server.

Here is an example. Our app lets a user input a URL and summarizes the content of the target page server-side using an AI SDK. A mischievous user passes localhost:3000 as the URL instead of a website they’d like to summarize. Your server will fire a request against itself or any other service running on port 3000 in your backend infrastructure. This is a severe SSRF vulnerability!

You’ll want to be careful when firing requests based on user inputs, especially server-side.

Number 9: Security Logging And Monitoring Failures

I wish we could establish a telepathic connection with our beloved Node.js server running in the backend. Instead, the best thing we have to see what happens in the cloud is a dreadful stream of unstructured pieces of text we name “logs.”

Yet we will have to deal with that, not only for debugging or performance optimization but also because logs are often the only information you’ll get to discover and remediate a security issue.

As a starter, you might want to focus on logging the most important transactions of your application exactly like you would prioritize writing end-to-end tests. In most applications, this means login, signup, payouts, mail sending, and so on. In a bigger company, a more complete telemetry solution is a must-have, such as Open Telemetry, Sentry, or Datadog.

If you are using React Server Components, you may need to set up a proper logging strategy anyway since it’s not possible to debug them directly from the browser as we used to do for Client components.

Number 8: Software And Data Integrity Failures

The OWASP top 10 vulnerabilities tend to have various levels of granularity, and this one is really a big family. I’d like to focus on supply chain attacks, as they have gained a lot of popularity over the years.

You may have heard about the Log4J vulnerability. It was very publicized, very critical, and very exploited by hackers. It’s a massive supply chain attack.

In the JavaScript ecosystem, you most probably install your dependencies using NPM. Before picking dependencies, you might want to craft yourself a small list of health indicators.

• Is the library maintained and tested with proper code?
• Does it play a critical role in my application?
• Who is the main contributor?
• Did I spell it right when installing?

For more serious business, you might want to consider setting up a Supply Chain Analysis (SCA) solution; GitHub’s Dependabot is a free one, and Snyk and Datadog are other famous actors.

Number 7: Identification And Authentication Failures

Here is a stereotypical vulnerability belonging to this category: your admin password is leaked. A hacker finds it. Boom, game over.

Password management procedures are beyond the scope of this article, but in the context of full-stack web development, let’s dive deep into how we can prevent brute force attacks using Next.js edge middlewares.

Middlewares are tiny proxies written in JavaScript. They process requests in a way that is supposed to be very, very fast, faster than a normal Node.js endpoint, for example. They are a good fit for handling low-level processing, like blocking malicious IPs or redirecting users towards the correct translation of a page.

One interesting use case is rate limiting. You can quickly improve the security of your applications by limiting people’s ability to spam your POST endpoints, especially login and signup.

You may go even further by setting up a Web Applications Firewall (WAF). A WAF lets developers implement elaborate security rules. This is not something you would set up directly in your application but rather at the host level. For instance, Vercel has released its own WAF in 2024.

Number 6: Vulnerable And Outdated Components

We have discussed supply chain attacks earlier. Outdated components are a variation of this vulnerability, where you actually are the person to blame. Sorry about that.

Security vulnerabilities are often discovered ahead of time by diligent security analysts before a mean attacker can even start thinking about exploiting them. Thanks, analysts friends! When this happens, they fill out a Common Vulnerabilities and Exposure and store that in a public database.

The remedy is the same as for supply chain attacks: set up an SCA solution like Dependabot that will regularly check for the use of vulnerable packages in your application.

Halfway break

I just want to mention at this point how much progress we have made since the beginning of this article. To sum it up:

• We know how to recognize an SSRF. This is a nasty vulnerability, and it is easy to accidentally introduce while crafting a super cool feature.
• We have identified monitoring and dependency analysis solutions as important pieces of “support” software for securing applications.
• We have figured out a good use case for Next.js edge middlewares: rate limiting our authentication endpoints to prevent brute force attacks.

It’s a good time to go grab a tea or coffee. But after that, come back with us because we are going to discover the five most common vulnerabilities affecting web applications!

Number 5: Security Misconfiguration

There are so many configurations that we can mismanage. But let’s focus on the most insightful ones for a web developer learning about security: HTTP headers.

You can use HTTP response headers to pass on a lot of information to the user’s browser about what’s possible or not on your website.

For example, by narrowing down the “Permissions-Policy” headers, you can claim that your website will never require access to the user’s camera. This is an extremely powerful protection mechanism in case of a script injection attack (XSS). Even if the hacker manages to run a malicious script in the victim’s browser, the latter will not allow the script to access the camera.

I invite you to observe the security configuration of any template or boilerplate that you use to craft your own websites. Do you understand them properly? Can you improve them? Answering these questions will inevitably lead you to vastly increase the safety of your websites!

Number 4: Insecure Design

I find this one funny, although a bit insulting for us developers.

Bad code is literally the fourth most common cause of vulnerabilities in web applications! You can’t just blame your infrastructure team anymore.

“

Design is actually not just about code but about the way we use our programming tools to produce software artifacts.

In the context of full-stack JavaScript frameworks, I would recommend learning how to use them idiomatically, the same way you’d want to learn a foreign language. It’s not just about translating what you already know word-by-word. You need to get a grasp of how a native speaker would phrase their thoughts.

Learning idiomatic Next.js is really, really hard. Trust me, I teach this framework to web developers. Next is all about client and server logic hybridization, and some patterns may not even transfer to competing frameworks with a different architecture like Astro.js or Remix.

Hopefully, the Next.js core team has produced many free learning resources, including articles and documentation specifically focusing on security.

I recommend reading Sebastian Markbåge’s famous article “How to Think About Security in Next.js” as a starting point. If you use Next.js in a professional setting, consider organizing proper training sessions before you start working on high-stakes projects.

Number 3: Injection

Injections are the epitome of vulnerabilities, the quintessence of breaches, and the paragon of security issues. SQL injections are typically very famous, but JavaScript injections are also quite common. Despite being well-known vulnerabilities, injections are still in the top 3 in the OWASP ranking!

Injections are the reason why forcing a React component to render HTML is done through an unwelcoming `dangerouslySetInnerHTML` function.

“

React doesn’t want you to include user input that could contain a malicious script.

The screenshot below is a demonstration of an injection using images. It could target a message board, for instance. The attacker misused the image posting system. They passed a URL that points towards an API GET endpoint instead of an actual image. Whenever your website’s users see this post in their browser, an authenticated request is fired against your backend, triggering a payment!

As a bonus, having a GET endpoint that triggers side-effects such as payment also constitutes a risk of Cross-Site Request Forgery (CSRF, which happens to be SSRF client-side cousin).

Even experienced developers can be caught off-guard. Are you aware that dynamic route parameters are user inputs? For instance, [language]/page.jsx in a Next.js or Astro app. I often see clumsy attack attempts when logging them, like “language” being replaced by a path traversal like ../../../../passwords.txt.

Zod is a very popular library for running server-side data validation of user inputs. You can add a transform step to sanitize inputs included in database queries, or that could land in places where they end up being executed as code.

Number 2: Cryptographic Failures

A typical discussion between two developers that are in deep, deep trouble:

— We have leaked our database and encryption key. What algorithm was used to encrypt the password again? AES-128 or SHA-512?
— I don’t know, aren’t they the same thing? They transform passwords into gibberish, right?
— Alright. We are in deep, deep trouble.

This vulnerability mostly concerns backend developers who have to deal with sensitive personal identifiers (PII) or passwords.

To be honest, I don’t know much about these algorithms; I studied computer science way too long ago.

The only thing I remember is that you need non-reversible algorithms to encrypt passwords, aka hashing algorithms. The point is that if the encrypted passwords are leaked, and the encryption key is also leaked, it will still be super hard to hack an account (you can’t just reverse the encryption).

In the State of JavaScript survey, we use passwordless authentication with an email magic link and one-way hash emails, so even as admins, we cannot guess a user’s email in our database.

And number 1 is…

Such suspense! We are about to discover that the top 1 vulnerability in the world of web development is…

Broken Access Control! Tada.

Yeah, the name is not super insightful, so let me rephrase it. It’s about people being able to access other people’s accounts or people being able to access resources they are not allowed to. That’s more impressive when put this way.

A while ago, I wrote an article about the fact that checking authorization within a layout may leave page content unprotected in Next.js. It’s not a flaw in the framework’s design but a consequence of how React Server Components have a different model than their client counterparts, which then affects how the layout works in Next.

Here is a demo of how you can implement a paywall in Next.js that doesn’t protect anything.

// app/layout.jsx
// Using cookie-based authentication as usual
async function checkPaid() {
  const token = cookies.get("auth_token");
  return await db.hasPayments(token);
}
// Running the payment check in a layout to apply it to all pages
// Sadly, this is not how Next.js works!
export default async function Layout() {
  // ❌ this won't work as expected!!
  const hasPaid = await checkPaid();
  if (!hasPaid) redirect("/subscribe");
  // then render the underlying page
  return <div>{children}</div>;
}
// ❌ this can be accessed directly
// by adding “RSC=1” to the request that fetches it!
export default function Page() {
  return <div>PAID CONTENT</div>
}

What We Have Learned From The Top 5 Vulnerabilities

Most common vulnerabilities are tightly related to application design issues:

• Copy-pasting configuration without really understanding it.
• Having an improper understanding of the framework we use in inner working. Next.js is a complex beast and doesn’t make our life easier on this point!
• Picking an algorithm that is not suited for a given task.

These vulnerabilities are tough ones because they confront us to our own limits as web developers. Nobody is perfect, and the most experienced developers will inevitably write vulnerable code at some point in their lives without even noticing.

How to prevent that? By not staying alone! When in doubt, ask around fellow developers; there are great chances that someone has faced the same issues and can lead you to the right solutions.

Where To Head Now?

First, I must insist that you have already done a great job of improving the security of your applications by reading this article. Congratulations!

Most hackers rely on a volume strategy and are not particularly skilled, so they are really in pain when confronted with educated developers who can spot and fix the most common vulnerabilities.

From there, I can suggest a few directions to get even better at securing your web applications:

• Try to apply the OWASP top 10 to an application you know well, either a personal project, your company’s codebase, or an open-source solution.
• Give a shot at some third-party security tools. They tend to overflow developers with too much information but keep in mind that most actors in the field of security are aware of this issue and work actively to provide more focused vulnerability alerts.
• I’ve added my favorite security-related resources at the end of the article, so you’ll have plenty to read!

Thanks for reading, and stay secure!

Resources For Further Learning

• An interactive demo of an SSRF in a Next.js app and how to fix it
• OWASP Top 10
• An SSRF vulnerability that affected Next.js image optimization system
• Observe React Server Components using Open Telemetry
• OpenTelemetry and open source Telemtry standard
• Log4J vulnerability
• Setting up rate limiting in a middleware using a Redis service
• Vercel WAF annoucement
• Mitre CVE database
• An interactive demo of a CSRF vulnerability in a Next.js app and how to fix it
• A super complete guide on authentication specifically targeting web apps
• Server form validation with zod in Next.js (Astro has it built-in)
• Sanitization with zod
• Secure statically rendered paid content in Next.js and how layouts are a bad place to run authentication checks
• Smashing Magazine articles related to security (almost 50 matches at the time of writing!)

This article is inspired by my talk at React Advanced London 2024, “Securing Server-Rendered Applications: Next.js case,” which is available to watch as a replay online.

(yk)

Today, we are very happy to announce our new book: Success at Scale, a curated collection of best-practice case studies capturing how production sites of different sizes tackle performance, accessibility, capabilities, and developer experience at scale. Case studies are from industry experts with guidance that stands the test of time.

Join Addy Osmani, your curator, as we dive into a nuanced look at several key topics that will teach you tips and tricks that may help you optimize your own sites. The book will also include short interviews with each contributor on what additional lessons, challenges, and tips they have to share some time after each case study was written.

High-quality hardcover. Curated by Addy Osmani. Cover art by Espen Brunborg. Print shipping in early March 2024. eBook now available. Pre-order the book.

Contents

Each section of the book is filled with case studies from real-world large-scale web applications and services, interviews with the people involved, and key takeaways to help you achieve the same success.

• Performance includes examples of measuring, budgeting, optimizing, and monitoring performance, in addition to tips for building a performance culture.
• Capabilities is about bridging the gap between native capabilities and the modern web. You’ll explore web apps, native apps, and progressive web applications.
• Accessibility makes web apps viable for diverse users, including people with temporary or permanent disabilities. Most of us will have a disability at some point in our lives, and these case studies show how we can make the web work for all of us.
• Developer Experience is about building a project environment and culture that encourage support, growth, and problem-solving within teams. Strong teams build great projects!

Who This Book Is For

This book is for professional web developers and teams who want to deliver high-quality web experiences. We explore dimensions like performance, accessibility, capabilities, and developer experience in depth. Success at Scale goes beyond beginner material to cover the pragmatic approaches required to tackle these challenges in the real world.

About the Author

Addy Osmani is an engineering leader working on Google Chrome. He leads up Chrome’s Developer Experience organization, helping reduce the friction for developers to build great user experiences.

Technical Details

• ISBN: 978-3-910835-00-9 (print)
• Quality hardcover, stitched binding, ribbon page marker.
• Free worldwide airmail shipping from Germany starting early March 2024.
• eBook available for download as PDF, ePUB, and Amazon Kindle.
• Pre-order the book.

Community Matters ❤️

Producing a book takes quite a bit of time, and we couldn’t pull it off without the support of our wonderful community. A huge shout-out to Smashing Members for the kind, ongoing support. The eBook is and always will be free for Smashing Members as soon as it’s out. Plus, Members get a friendly discount when purchasing their printed copy. Just sayin’! ;-)

More Smashing Books & Goodies

Promoting best practices and providing you with practical tips to master your daily coding and design challenges has always been (and will be) at the core of everything we do at Smashing.

In the past few years, we were very lucky to have worked together with some talented, caring people from the web community to publish their wealth of experience as printed books that stand the test of time. Heather and Steven are two of these people. Have you checked out their books already?

No matter what your stance is on them, no matter what your perspective is on data protection, web advertisement, setting cookies, EU’s General Data Protection Regulation (GDPR), and privacy preferences, cookie consent widgets (or “cookie banners”) are a reality on the web today.

For this reason, it is worth looking into how accessible and usable these banners are or can be. They have become, for better or worse, a component of the majority of today’s websites. Even more, cookie banners are often the first thing a user encounters. And, of course, they are part of every site of a webpage once they are implemented.

Sometimes, cookie banners are a technical necessity because of the page’s feature set or because of advertisements on the page. Even more often, cookie banners are not built by the front-end team but are a ready-made solution, like UserCentrics or others.

Before I explain why the cookie banner deserves special attention regarding its accessibility, let’s quickly explain how the current gold standard of web accessibility, Web Content Accessibility Guidelines (WCAG) Version 2.1, works.

WCAG consists of principles, guidelines, and success criteria. The latter are testable steps to check against a webpage. For example:

• “Is the main language of the document set?”
• “Does this non-text content have a suitable text alternative?”
• “Is it perceivable where my focus is when I’m using the web presence with the keyboard (or another tech that emulates keyboard presses)?”

You may have noticed that these are “yes or no” questions. Accordingly, this means that the final verdict of any given success criterion is either “pass” or “fail.”

Additionally, conformance to WCAG, as defined by the W3C (the governing body of the Web), means that none of its success criteria is allowed to “fail” when the whole document needs to be conformant:

“Conformance to a standard means that you meet or satisfy the ‘requirements’ of the standard. In WCAG 2.0, the ‘requirements’ are the Success Criteria. To conform to WCAG 2.0, you need to satisfy the Success Criteria, that is, there is no content which violates the Success Criteria.”

— W3C Working Group Note

No nuance here. Going back to our cookie consent interface, this means that the banner (or any other component) alone has the potential to negatively affect the WCAG conformance of an entire web project.

WCAG conformance could be a big legal deal for many websites, whether part of the public sector in the European Union or the United States, as it is considered to fall under non-discrimination or market access laws or overall human rights to access to information. Webpages frequently must adhere to directives and regulations that directly or indirectly refer to WCAG, often its newest version, and conformance to its level AA standards. Therefore, all the following WCAG criteria are viewed through this lens, being fully aware that they are only a mere starting point when it comes to true web accessibility. On top of that, cookie consent interfaces are implemented on every subpage of a website, consequently harming accessibility and conformance throughout an entire website.

So, in order to not let a faulty cookie banner interface drag down your page’s conformance with accessibility laws and, more importantly, not exclude users from accessing and exercising their rights, let’s list what to look for, what to configure, and what to build properly in the first place.

Contrast Errors

This is especially relevant when it comes to important controls such as the setting of cookies or the overall acceptance of the recommended cookie set. It is crucial that form controls and text can be sufficiently perceived. Unsurprisingly, a solid contrast is also important for WCAG in general. Namely, in success criteria 1.4.3 and 1.4.11, both define contrast boundaries.

What To Do

When you are using a ready-made cookie management solution, try to influence the colors (if possible, potentially in your cookie vendor’s settings) and make sure interactive controls have sufficient color contrast.

Additionally, if your website relies on a dedicated contrast mode for WCAG conformance, check whether it extends to (or influences) the cookie management interface. I have seen cases in my accessibility auditor practice where this was not considered, and an inaccessible (often branded) color combination was used in the cookie interface, thinking the contrast mode takes care of every color-related violation. But the contrast setting of the website did not affect the third-party cookie banner due to it being, well, third-party and loaded from external sources or after the contrast mode had done its work, resulting in a “Fail” on WCAG’s contrast-related success criteria.

Pseudo Buttons

Another cookie banner issue can be one thing that is, unfortunately, an error pattern that you can find outside of cookie management: divs or spans with click events posing as links or buttons. These controls may be styled like buttons but lack the semantic information of a button.

On top of that, these controls usually aren’t keyboard focusable. Hence, many serious barriers and WCAG violations are occurring all at once. If we were about to imagine the most “pseudo” button, e.g., a div with a click handler, this would at least violate success criteria 2.1.1 (Keyboard), because it is neither reachable nor “activatable,” and 4.1.2 (Name, Role, Value) because it doesn’t “introduce” itself as a button and lacks a programmatic label.

What To Do

The easiest thing to do, assuming you have built the cookie management interface yourself, is to replace those above-mentioned pseudo buttons with real <button> elements because it provides semantics, focusability, and even keyboard event handlers for free. But even if we don’t talk literally about buttons, the pattern is the same: check your cookie prompt for interactive elements that are built with elements that are only styled to look like “the real thing” but consist of non-semantic divs and spans. This is a red flag for you to implement native interactive elements, like a, button, or input instead.

The situation gets a lot tougher, of course, when these semantic errors are in a third-party script and are, therefore, beyond your direct influence and control. Understandably, we have to leave the engineering side of things and start to dive into politics of some sort. If you work within an organization where the decision of cookie management infrastructure is outside your control, you have to escalate matters to your supervisors and managers (especially, but not only when your web projects have to adhere to accessibility laws).

Three abstract steps have to happen:

1. Your organization has to become aware of the barrier and potential legal risk — “up” to the powers that have the influence to change technical decisions like these.
2. As a consequence, the vendor that provided the faulty cookie banner has to be contacted about the issue.
3. A form of pressure should be applied by your organization — not just for your own sake but also regarding all the other web pages where the faulty cookie banner negatively influences accessibility and conformance.

In a possible fourth step, your company or agency should reflect on its vending process for third-party services and the HTML (and possible barriers) that come with it.

Unlabeled Form Fields

When you think about it, the main user control that one could imagine for cookie management widgets is a form control: You can select which set of cookies you are willing to accept by interacting with checkboxes in a form element. And, of course, it is important that checkbox inputs are built in the correct way.

Alas, that is not always the case. While a checkbox and its label may visually appear adjacent, the checkbox can lack a programmatic label. This adds unnecessary confusion and barriers to the interface and also a failure of success criterion 1.3.1 when you look into the web accessibility standard.

What To Do

The most solid strategy to connect form inputs with their corresponding labels is to:

1. Use a label element for the label (obviously).
2. Establish an id on the respective input you want to label.
3. Add a for attribute, filling it with the control’s id you created in the last step.

This also works for inputs of all types, like textareas and selects. Here’s an example of how a properly labeled checkbox could look:

<input type="checkbox" id="marketing-cookies" />
<label for="marketing-cookies">Accept marketing cookies</label>

If you can’t directly influence the HTML of the cookie banner’s code, the situation is comparable to the situation around pseudo buttons. Make sure that necessary pressure is applied to your cookie service provider to fix the problem. All of their customers will thank you for it, and even more so the people who visit their sites.

Broken Dialog Semantics (Or None At All)

Quite a few cookie banners are actually cookie dialogs, and of the modal kind. Modal, in the context of a dialog, means that such a window blocks everything but itself, leaving only itself accessible. That is, at least, the theory. But quite some cookie management dialogs do “want to be as aggressive,” presenting as a modal part of the interface but have no according semantics and behavior, which would violate WCAG success criterion 4.1.

What To Do

Up until recently, the recommendation was to build a dialog with WAI-ARIA roles and states and implement focus management yourself (or use Kitty Giraudel’s great a11y-dialog) component).

But the situation has (mostly) changed for the better. Lately, the native <dialog> element has matured to the point where it’s being recommended in most contexts as long as it is used reasonably. A great win for accessibility, in my opinion. The past way of building (modal) dialogs had so many moving parts and factors (roles, states, focus behaviors) to think about and implement manually that it was quite difficult to get it right. Now creating a dialog means using an aptly-named HTML element (and initializing it with .showModal() if you think the cookie dialog needs to be interface-blocking).

What I’ve written so far is, of course, also valid when you cannot influence a third party’s code, and what I wrote earlier about comparable situations and potential cookie consent barriers is valid as well. If you detect errors in the third-party script you are implementing (such as no focus trapping, no dialog role, no aria-modal="true" — and if everything else points towards “modalness”), escalate things internally and educate the decision-making powers about web accessibility consequences. Maybe even educate the third-party developers that things concerning modals have gotten a lot better recently.

Cookie Banners Are Hard To Find In The First Place

There are three typical places where you can usually find cookie consent interfaces, at least visually:

1. As a modal dialog, i.e., in the middle or — more rarely — corners of the viewport;
2. On top, sometimes in a fixed manner;
3. At the bottom of the viewport, sometimes also somewhat positioned in a fixed way.

But what matters way more for some people is how easy it is to find, should they go on a hunt for it. A great way of presenting this very problem is a presentation that accessibility specialist Léonie Watson gave some time ago. Léonie is a seasoned screen reader user, and her presentation showcases a bunch of webpages and how the placement and “findability” of cookie banners influence the screen reader experiences, particularly as it is related to privacy. Hampering the ability to find important content in a document can, for example, negatively affect WCAG 1.3.2 (Meaningful Sequence).

What To Do

In Léonie’s presentation, the best practices for cookie notice findability become very clear, especially in the last example:

• Place the banner preferably at the top of the document.
• Use a headline in the cookie banner and make it either visible or visually hidden to help screen reader users “get a grasp about the webpage” and allow them to navigate by headings.
• Build a bridge back to proper dialog semantics by making sure that if a dialog is meant to be the “exclusive” part of the interface, it uses appropriate semantic and state descriptions (see above for details).

When we’re talking about changing third-party code, I reckon you know the drill by now. Try to influence this code indirectly on the “political level” because direct control is not possible.

Conclusion

Hopefully, two things emerged while reading this article:

1. Awareness of the issue, namely, that an often unloved stepchild interface element has the potential to make it harder for some people to manage their privacy settings and, on top of that, to even pose a legal risk.
2. A sense of how you can possibly remediate barriers you encounter when working with a cookie management banner. The direct way is described in a certain detail in the details I provided earlier and often has to do with code, styling, or overall education on how to prevent this in the future. The indirect way leads to a path of either setting the consent interface up properly or influencing the inner and outer politics of your vendor scripts. And again, there is the aspect of educating everyone involved. This time, structured information may be aimed at the powers that be in your organization, showing them that their choice of service providers may have unintended consequences.

But regardless of whether you and your team manage to fix accessibility bugs directly or indirectly in your cookie consent interfaces, you can see their ubiquity and component architecture as an advantage. By getting the accessibility right in one place, you influence many other pages (or even foreign websites) for the better.

If you want to extend your horizon regarding the user experience side of cookie banners and learn about how you can actually turn privacy settings into a pleasant and respectful involvement with at least EU laws, please proceed to Vitaly’s smashing read, “Privacy UX: Better Cookie Consent Experiences”.

Further Reading On SmashingMag

• “State Of GDPR In 2021: Cookie Consent For Designers And Developers”, Danny Bluestone
• “A Guide To Keyboard Accessibility: HTML And CSS (Part 1)”, Cristian Díaz
• “Keys To An Accessibility Mindset”, Daniel Yuschick
• “Using Automated Test Results To Improve Accessibility”, Noah Mashni and Mark Steadman

(gg, yk, il)

Many apps have some kind of user-specific information or data that is supposed to be accessed by a certain group of users and not by others. With these sorts of requirements comes a demand for fine-grained access handling. Whether for security or privacy reasons, dealing with sensitive data is an important topic for any app. Big or small, nobody wants to be on the wrong side of a data leakage scandal. So let’s dive in on what it means to handle sensitive or confidential information in our apps.

Take It Seriously

Regardless if you’re requesting access on Twitter, a bank, or your local library, identifying yourself is a crucial first step. Any sort of gateway needs a reliable way to verify if an access request is legitimate.

“Identity theft is not a joke.”
— Dwight Schrute

On the web, we encapsulate the process for identifying a user and granting them access as Auth, which stands for two related but distinct actions:

• Authentication: the act of confirming a user’s identity.
• Authorization: granting an authenticated user access to a resource.

It is possible to have authentication without authorization, but not the other way around. The strategy to implement authorization at a data management level can be loosely referred to as Row-Level Security (RLS), but RLS is actually a bit more than this. In this article, we will take a step deeper into managing sensitive user data and defining access roles to a user base.

Row-Level Security (RLS)

A ‘row’, in this case, refers to an entry in a database table. For example, in a posts table, a row would be a single article, check this json representation:

{
    "posts": [
        {
            "id": "article_23495044",
            "title": "User Data Management",
            "content": "<huge blob of text>",
            "publishedAt": "2023-03-28",
            "author": "author_2929292"
        },
        // ...
    ]
}

To understand RLS, each object inside posts is a ‘row’.

The above data is enough for creating a filter algorithm to effectively enforce row-level security. Nonetheless, it’s crucial for scalability and data handling that such relationship is defined on your data layer. This way, any service that connects to your database will have all the required information to implement its own access-control logic as required. So for the above example, the schema for the posts table would roughly look like the following:

{
    "posts": {
        "columns": [
            {
                "name": "id",
                "type": "string"
            },
            // ... other primitive types
            // establish relationship with "authors"
            {
                "name": "author",
                "type": "link",
                "link": "authors"
            }
        ]
    }
}

In the above example, we define the type of each value in our posts database and establish a relationship to the authors table. So each post will receive the id of one author. This is a one-to-many relationship: one author can have many posts.

Of course, there are patterns to define many-to-many relationships as well. Take, for example, a team’s backlog. You may want only members of a certain team to have access. In such case, you can create a list of users with access to a specific resource (and thus being very granular about it), or you can define a table for team, and thus connecting a team to multiple tasks, and a team to multiple users: this pattern is called a junction table and is great for defining scoped access within your data layer.

Now we understand what authorization is and looks like in a few cases. This should be enough to design a mental model for defining access to our data. We understand that in order to use the granular access to our data effectively, our app must be aware of which user is using that particular instance of the app (aka who’s behind the mouse).

Authentication

It is time to set up a reliable and cost-effective authentication. Cost-effective because it is counter-productive to re-authenticate the user on every request. And it increases the risk factor of attacks, so let’s keep auth requests to a minimum. The way our app stores the user credentials to re-use in a defined lifecycle is called a session.

There are multiple ways of authenticating users and handling sessions. I invite you to check Eric Burel’s article on “Authentication in Websites: A Banking Analogy”. It’s a great and thorough explanation of how authentication works.

From this moment on, let’s assume we did our due diligence: username and password are securely stored, an authentication provider is able to reliably verify our user’s identity and returns a session, which is an object carrying a userId matching our user’s row in the database.

Connecting The Dots

So now that we have established what it means and the requirements each moving piece brings in order to get it working, our goal is the following:

1. Authentication
   The provider performs user authentication, the library creates a session, and the app receives that as a payload from the auth request.
2. Resource request
   Authenticated User performs request with resourceId; the app takes userId from session.
3. Granting access
   It filters all resources from the table to only the ones owned by userId and returns (if it exists) the one with resourceId.

With the above mental model defined, it is possible to any sort of implementation and properly design your queries. For example, on our first defined schema (posts and authors), we can use filters on our fetching service to only provide access to the results a user should have:

async function getPostsByAuthor(authorId: string) {
    return sdk.db.posts
        .filter({
            author: authorId
        })
        .getPaginated()
}

This contrived snippet is just to exemplify a bare-bones RLS implementation. Maybe as a food-for-thought so you can build upon it.

Conclusion

Hopefully, these concepts have offered extra clarity on defining access management to private and/or sensitive data. It’s important to note that there are security concerns before and around storing such kind of data which were beyond the scope of this article. As a general rule: store as little as you need and provide only the necessary amount of access to data. The least sensible data going over the wire or being stored by your app, the lesser the chance your app is a target or victim of attacks or leaks.

Let me know your questions or feedback in the comment section or on Twitter.

(yk)

There is a strange ritual that web developers around the world have been perpetuating from the dawn of computers to modern days. Without this ritual, the web application cannot really exist. It is not “ready to go live,” a web developer would say. This ritual is the implementation of authentication.

You may have been through this ritual multiple times. But do you really understand what is happening?

This article is my attempt at making this ritual less obscure. It shouldn’t have to take a wizard to implement a good authentication system. Just a good banker!

Your Password Matched. Now What?

Authentication is the process of attributing a given request to a unique entity, usually a person. It then enables authorization, the process of allowing (or not) said person to access a resource, be it a web page or an API endpoint.

I will focus on the part I’ve always found the most difficult to understand: what happens after the user just submitted the login form and their credentials were confirmed as valid by the backend.

  app.post("/login", async (request, response) => {
    const { body } = request;
    const user = await getUserByCredentials(body.username, body.password);
    if (user) {
      // NOW WHAT???
    }

Your typical login endpoint. This article is about what happens afterward.

We are going to talk about tokens, CORS, cookies, and headers. Those concepts are sometimes quite abstract. To make them more real, I’ll take a shot at a banking analogy.

Authentication Relies On Tokens Like Banks Rely On Money

Let’s consider two common patterns:

• JSON Web Tokens aka JWT.
  It’s an open standard. You’ll find useful resources at the bottom of this article if you want to go further.
• Session-based authentication.
  It’s a pattern more than a standard. So the implementation may vary, but I’ll try to give an overview of how it works.

As a front-end developer, you may have no control over this design decision, so it’s important that you understand how both work.

As a full-stack or backend developer, you have control over this choice, so it’s even more important that you understand what you are doing. I mean, it’s kinda your job :)

A JWT Is Like A Banknote

With JSON Web Token-based authentication (JWT), during the login step, the server will pass the client a token that contains encrypted information, a “claim.” When the client makes a request, they pass this token alongside the request.

The server verifies that this token is indeed a valid token that it itself generated earlier. Then the server reads the token payload to tell who you are, based on a user ID, username, email, or whatever the payload is. Finally, it can authorize or refuse access to the resource.

Think of tokens as banknotes. A banker can print new banknotes like a server can create tokens. Money lets you buy things, or if I rephrase, it lets you access paid goods and services. The JWT lets you access secure services the same way.

If you want to use your money, either at the bank to credit your account or in a store, people will double-check that your banknotes are real. A real banknote is one that has been created by the bank itself. When the banknote is confirmed to be legit, they can read the value on it to tell how many things you can buy.

JWTs work the same, the server can verify that they indeed created this token earlier, so it’s a real one, and then they can read the payload it contains to confirm who you are and what your permissions are.

Revoking a token can be hard. The server has to maintain a list of blacklisted tokens that are no longer valid, like the bank would need to list robbed banknotes’ unique numbers.

Session-Based Authentication Is Like A Credit Card

From the front-end developer standpoint, the main difference between JWT and session-based authentication lies in what the server will return in case of a successful login.

In session-based authentication, instead of having a claim full of user information, you just get a “session id.”

From the backend developer’s standpoint, session-based authentication involves an additional database collection or table to store the sessions. A session is just an object very similar to the JWT, which can contain, for instance, a user id and an expiration date.

This means that the server keeps a lot of control over the authentication. It can log a user out by removing the session from the database, whereas it cannot destroy JWTs, since it doesn’t store them.

Session-based authentication is comparable to credit cards. You go to the bank to open your account. The banker will check your id and issue a credit card with an expiration date. This is the authentication process.

The credit card itself has barely any value: it’s just plastic and a microchip whose purpose is to uniquely identify your account. It can easily be revoked by the bank, usually after a certain period of time, but also when you declare your card stolen.

When you want to pay for something, the store will communicate with your bank to check that your credit card is valid and has a positive balance.

When you access an authenticated resource, the server will check if the session id matches an open session and then if this session matches a user that is authorized to access this resource.

Tokens Are As Precious As Banknotes And Credit Cards. Don’t Get Them Stolen!

It’s easy for companies and banks to store money. They have those big reinforced-steel safes with alarms and sometimes even dragons. Likewise, it’s easy for servers to talk securely to one another.

But websites are like private persons. They can’t have dragons, so they have to imagine specific approaches to store their precious tokens.

Where Do You Store Your Money?

Whatever the pattern you pick to authenticate users, JSON web tokens and session ids are precious things. In the remainder of this article, I will treat them similarly, apart from a few technical subtleties. I’ll call either of them the “authentication token,” the thing you show the server to prove that you are authenticated, like money for paying stuff.

Since you have to pass the token alongside every request you make to the server — you need to store it somewhere on the client side.

Here, I’d like to focus more specifically on the case of websites, where the web browser is in charge of sending the requests. You’ll find many articles and documentation in the wild that do not clarify whether they apply to server-server communication or browser-server. The latter is a trickier, uncontrolled context that deserves its own specific patterns.

So, the banker just handed you either a new credit card or a bunch of banknotes. Where do you stash them? In a compartment concealed in the heel of your shoe, in your mailbox, beneath your mattress, or maybe, in your fancy fanny pack?

Not All Requests Are Born Equal: Loading A Web Page vs. Calling An API

We can divide authenticated content into two categories: web pages and API endpoints. From a technical standpoint, both are server endpoints. The former returns HTML content or files, the latter — any kind of content, often JSON data. They differ in the way they are consumed.

Web pages are accessed via a web browser that triggers GET requests automatically, depending on the current URL.

APIs are called using JavaScript code. This reliance on APIs and JavaScript is at the core of the Jamstack, REST, and Single-Page-Application philosophies.

Thus, let’s differentiate two use cases:

1. You want to secure access to the web page itself. Meaning the user can’t even see the structure of your page or any HTML.
2. You want to secure calls from your web page to your API. Anyone can see your web page structure, but they can’t get any data.

If you combine both, unauthenticated users can’t see the page structure and can’t get any data from your API.

Use Case 1: Securing A Web Page Access With Cookies

I’ll cut it short, to secure web page access, you have no other choice but to use cookies, ideally HTTP-Only, secure cookies.

From a technical standpoint, a cookie is a piece of data that should be sent alongside every request to a specific domain. They are part of the HTTP protocol.

That’s a pretty broad scope. When a website cookie popup differentiates “essential cookies” from other cookies, those essential cookies are often dedicated to authentication. The others are just a way to keep track of the product pages you’ve visited to later shove them in your face to the point of literally optimizing the billboard displays in your neighborhood as if you really, really wanted to buy two freezers within the same month.

Setting a cookie is possible in both directions. Client-side JavaScript can set cookies to be sent to the server. That’s how advertisement tracking works (though GDPR is forcing advertisers to use server-side logic more and more these days).

But the server can also tell the browser to store a cookie using the Set-Cookie header in the response.

This convenient feature lets the backend set what we call “HTTP-only” cookies: cookies that cannot be read by JavaScript. And “secure” cookies are only set through HTTPS for better security.

Important note: Using HTTP-only cookies prevents XSS attacks (script injection to steal the token) but not CSRF attacks (forging requests on behalf of your authenticated users, basically impersonating them). You need to combine multiple strategies to parry all possible attacks. I invite you to read more on these topics if you manage a non-trivial website.

During the authentication process, the Set-Cookie header should be used in response to the “login” request so the client becomes authenticated via a token.

Cookies are like a wallet. You can store fidelity cards in it and a picture of your cat, but also duller things, like your money. It’s in your pocket or your handbag, so it’s hard to access for other people (HTTP-only, no JavaScript access) if you don’t take it out in unsafe places (secure). Yet, you always have your token with you if needed.

Browsers rely a lot on cookies, and they send them automatically alongside every request they trigger as long as the domain and path match, so you don’t have to worry about them. If you carry your wallet with you anytime you go out, you’re always ready to pay.

Use Case 2: Securing API Calls. The Cookie Scenario

Now, let’s authenticate calls to our API, which feeds the website with data. The easiest option would be to use a cookie, as we did in the previous section, to secure web pages. But contrary to the web page, there is also another option — using web storage. Which one is the most suited is debatable. Let me start with cookies.

The cookie that contains the auth token is sent automatically alongside requests triggered by the browser when you navigate a website or when you submit a form.

But requests triggered by JavaScript code do not fall into this category. When you trigger a request programmatically, you need a bit more configuration to send cookies.

Remember that the cookie should be HTTP-only, so you can’t read the cookie content with JavaScript nor inject your token in the Authorization header. You could do that with a cookie that is not HTTP-only, but then using a cookie doesn’t make sense in the first place.

All existing libraries capable of sending requests have to be based on the browser JavaScript API aka the “Browser Object Model,” which means either XMLHttpRequest or fetch. You have to rely on their configuration to enable sending HTTP-only cookies alongside the request.

In the case of fetch, this option is named credentials. Hopefully, for you, fellow front-end developers, it’s very easy to use.

fetch(
"http://localhost:8000/account/signup", 
{ 
  method: "POST",
  body: JSON.stringify({username:"foo", password:"barbar"}), 
  // This will include credentials (here, cookies) in the request
  credentials: "include"
})

With XHR, the option is named withCredentials and behaves similarly.

Note: Your server must also confirm that it indeed accepts credentials by setting the Access-Control-Allow-Credentials header to true in the responses. This option is symmetrical. It controls both sending cookies to the server, but also accepting the Set-Cookie header from it. So don’t forget to set it to true for the call to the login endpoint.

If the credentials option is not set and the server answers with a Set-Cookie header… the browser will silently ignore this attempt to set the cookie without even a warning. When this happens, start by double-checking your credentials option and then CORS if applicable (see next section).

Oops, I’ve almost forgotten your banking analogy! It’s a tough one but let’s say that you probably have your credit card or a few banknotes always there in your wallet. But some services expect a prepaid card, and you may need to explicitly remember to take them with you. That’s the best analogy I could achieve. Comment if you can do better!

Return Of Use Case 2: Securing API Calls. The Web Storage Scenario

Remember, I said earlier that for API calls, you have the choice to either store the token in an HTTP-only cookie, like you would do to secure a web page, or to store the token in the web storage. By web storage, we mean either sessionStorage or localStorage.

I’ve seen and used both approaches, cookies and web storage, but I sincerely don’t have the expertise to assert that one is better than the other. I’ll stick to describing how it works so you can make an enlightened decision.

Codewise, they are much easier to use than HTTP-only cookies because web storage is available to JavaScript code. You can get the token from the login response payload, store it in localStorage, and insert it into the Authorization header in later requests. You have full control of the process without relying on some implicit browser behavior.

headers: {
    “Authorization”: `Bearer ${window.localStorage.get(“auth_token”)}`
}

Web storage is immune to certain attacks, namely Cross-Site Request Forgery. That’s because, contrary to cookies, they are not automatically tied to requests, so an attacker will have trouble forcing you to give your token. Think of a piggy bank. You can’t just get money out of it. It’s, however, very sensitive to script injection, where an attacker manages to run JavaScript and reads the token.

Therefore, if you go the web storage way, you’ll want to explore strategies that prevent XSS attacks from happening in the first place.

JWT Specifics: What Information They Should Contain And How To Handle The Refresh Token

You should avoid storing critical information in a JSON web token, even when using HTTP-only cookies. If someone still manages to steal and decrypt the token, they will be able to impersonate the user, but at least they won’t be able to get much information from the token itself.

You can afford to lose a five-dollar banknote; it’s more problematic to lose a $5,000 banknote. By the way, if such a thing as a $5,000 banknote exists, please send me a specimen to prove it. Thank you.

JWTs are meant to be short-lived, like banknotes, which would wear quickly when used a lot. Since they are hard to revoke, giving them 5-minute lifetime guarantees that even a stolen JWT token cannot be a big problem.

Since you don’t want the user to log in again every 5 minutes, the token can be combined with a longer-lived refresh token. This second token can be used to get a new JWT.

A best practice is to store the refresh token in a cookie on a very specific path like /refresh (so it’s not sent with all requests, contrary to the JWT) and to make it a one-time-only token.

A refresh token is like bringing your ID card to get more money from your bank account. You don’t do that often, and you probably want to store your ID even more safely than your banknotes.

CORS Is How Shops Communicate With Their Bank

Formally, CORS (Cross-Origin Resource Sharing) is a protocol used by servers in order to instruct browsers to selectively relax some of the Same-Origin Policy (SOP) restrictions on network access for some trusted clients.

Less formally, the server can tell your browser that the request you send is legit based on the request’s origin. If the server says it’s not legit, the browser won’t send the request.

For instance, https://www.foobar.com, https://qarzed.fake, and http://localhost:8080 are possible origins. The port and protocol matter.

An API hosted on https://api.foobar.com will most often allow requests only from https://www.foobar.com and https://mobile.foobar.com.

I will focus here on API calls triggered with fetch in JavaScript, and only if the API lives in another Web origin. CORS also affects more advanced use cases like embedding images from another website in your own website, iframe, navigating from one site to another, and so on, but that’s beyond the scope of authentication per se.

Websites are like stores. People can enter them and try to buy things. But the store also has its own complicated internal back office logic. Namely, it relies on a bank to process payments.

Any shop owner can try to connect to any bank. Any website can try to connect to any API. But the payment won’t be processed by the bank if they don’t have a company account there.

Likewise, if you host the website on www.foobar.com and the API on www.foobar.com/api, there is one origin only www.foobar.com. No need for CORS because there is no “cross-origin” call involved. You’ll need to bother about CORS if, instead, you host the API on api.foobar.com. We have two distinct origins, api.foobar.com and www.foobar.com.

This also applies to using different ports on localhost during development. The IP 127.0.0.1 is even different from localhost, as origins are compared directly as strings and not based on their actual IP address.

Some big companies, like car sellers, have their own internal bank. It’s easier for them to communicate with this bank if it’s literally located in the same office. They are like full-stack monolithic applications.

But if this internal bank is located in a different country (API separate from the website), this doesn’t work anymore. It could as well be a traditional bank with multiple clients (a third-party API). CORS is needed either way.

CORS is a concept specific to browser-server communication. When a server talks to another server, the Same-Origin Policy doesn’t apply. The API may either block those requests with no origin completely or, on the contrary, accept them, or filter them based on IP and so on.

Banks have specific patterns of communicating with each other that are different from how they communicate with their client companies.

A Polite Cross-origin Discussion Between A Server And A Client, Using Preflight Requests

Here’s a more concrete vision of what happens during a cross-origin request and how the server response headers should be set.

From the browser standpoint, whenever a website sends a “non-simple” request to an API, the browser will first send a preflight request. It’s a request to the same URL but using the OPTIONS method, which is dedicated to checking whether you can actually consume this API endpoint.

Fetching JSON data falls into this “non-simple” category in standard browsers, as well as calling your login endpoint if the auth API doesn’t live on the same domain as your website, so this step is extremely important.

The server receives this request, which includes an Origin header. It checks its own list of accepted origins. If the origin is allowed, it will respond with the right Access-Control-Allow-Origin header.

This server header should match the website’s origin. It can also be a wildcard * if the API is totally open, but that works only for requests without credentials, so forget about this if your API requires authentication.

If the response header doesn’t match, the browser will trigger an error at this point because they already know that they can’t call this API. Otherwise, they will keep going and send the actual request.

A store (the API) can decide to accept only certain types of credit cards. Some stores accept only their own card, and they don’t even use CORS (same-origin scenario). Some stores accept only Visa or Mastercards (cross-origin scenario). The cashier (the browser) won’t let you pay if they are told your card is not accepted by the store.

There are a few other Access-Control headers that provide more information, like accepted methods or headers. Allow-Origin is the most basic one, as well as Allow-Credentials, so cookies and Set-Cookie headers are properly enabled when using fetch.

Access-control-allow-origin: https://www.foobar.com
Access-control-allow-credentials: true

Expected response headers for an authenticated call (or the “login” form submission) if the website lives on https://www.foobar.com and the API is on another domain or port.

Note: The server specifies the CORS policy, and the browser enforces it. If suddenly everyone used an unsafe browser run by pirates, they could swarm your server with forged requests. Hopefully, people usually don’t use a shady browser they found in a USB key itself found lying on the ground of a dark alley at 3 am.

The header things and the browser behavior are just politeness. The browser is an excessively polite software. It will never spam the server with requests that will be rejected anyway. And it won’t talk to rude servers that don’t set the proper headers. That’s why the Accept-Control-Allow-Origin response header must be correctly set for cross-origin requests to work.

Before going as far as aiming a customer with a shotgun, the banker will probably tell you, “sir, you entered the wrong bank office,” and the customer will probably answer, “oh, sorry, my mistake. I will leave immediately!”.

Cookies Or Not Cookies: The SameSite Option

Setting the credentials attribute in your fetch request is not enough for sending the cookie, plus it doesn’t apply to pages, only to programmatic requests. The SameSite attribute of cookies lets you configure when a cookie should be sent, depending on the current site.

The concept of “Site” is slightly different from the concept of domain:

• https://api.foobar.com and https://www.foobar.com are 2 different domains.
• They are, however, the same site. Nowadays, the scheme matters, too (HTTP vs. HTTPS).
• There is an exception to this rule, e.g., in multi-tenant architectures: foo.github.io and bar.github.io will be different sites despite having the same domain, and the server owner can configure that to their will.

If the Access-control response headers are correctly set, and the preflight request succeeds, the browser will proceed with the actual request.

But if SameSite is not set with the right value in the cookie containing the authentication token, this cookie won’t be sent to the server. When loading a page, you will be considered unauthenticated by the server just because it literally doesn’t have your cookie.

The SameSite: Lax value is usually the most appropriate. It will only send cookies to the same site but also when the user comes from another site and is redirected to your page (only for top-level navigations). This is the current default in most browsers.

With SameSite: Lax, the website www.foobar.com will automatically include the authentication cookie when it sends requests to api.foobar.com, which is part of the same site. Authentication will work as expected.

Sec-Fetch Helps Detecting Cross-origin Requests But Are Not Standard Yet

Related to CORS, you may also hear about the Sec-Fetch request headers. They provide information about the request context to the server. For instance, if Sec-Fetch-Site is same-origin, there is no need to check the Origin header. It’s slightly more intuitive than manually checking the request origin against predefined rules. If it’s same-site, you still need to add the required CORS headers in the response because the site and the origin are different concepts.

However, those headers are not sent by all browsers. Namely, Safari doesn’t support them at the time of writing. Therefore, they can only act as a secondary solution to identify a request’s origin.

A bank can identify you via your passport, but not everybody has a passport, so you will always need to resort to a good old ID card. Passports are still a cool way to identify people when available. That’s what the Sec-Fetch headers are.

Keep in mind that attackers may trick users into sending requests they don’t want to, but they still have no control over which browser will send the request. Thus, Sec-Fetch headers are still useful despite being supported only by some browsers. When they are present, they are a reliable piece of information.

More broadly, cross-origin and cross-site security patterns are based on the fact that users are using legitimate browsers that should never try to voluntarily mislead servers about the origin. Beware of more elaborate attacks, such as subdomain takeover, that can still bypass these security measures.

You may think, what about servers, then? Server requests are able to spoof a request origin, effectively bypassing CORS. Well, that’s right: CORS is a browser-server security mechanism; it simply doesn’t apply to server-server calls. Someone can build an API that calls your own API and serve the result to another website. True.

However, the thief will have to host a server, which isn’t free, so they will pay actual money for this API call, and the server IP may be traced back to them. It’s way less attractive than forcing browsers to send the request directly to your API, which would be free and trace back to the first victim, your user, and not to the attacker. Thanks to CORS, this worst-case scenario is not possible.

No, We Don’t Take Checks: Don’t Confuse A Web Page Access And An API Call

If you’ve stumbled upon this article, you have probably read a lot of documentation before that. You might have encountered various patterns that seem very different from one another and may be very different from what I’ve been describing so far.

I’d like to point out that this article is about authentication for websites. This is, I believe, the root of all confusion. First, the requests are sent by a browser, not by another server. Then, a page of a website can be accessed via the URL bar, and the website can send calls to an API via JavaScript; that’s two very different situations.

We have detailed the technical differences between each scenario earlier in this article. But let’s recap again to better understand the most common sources of confusion.

Using Browser Storage (Session, Local, And Friend) Is Like Using Your Mailbox As A Piggy Bank

Setting your token in localStorage is like storing money in your physical mailbox. Well, technically, it’s safe. It’s even used to temporarily store your mail. Session storage is similar; it’s just like getting your mail every day instead of letting it rot for weeks. Be nice to the postman!

But either way, the storage is accessible to JavaScript. It’s not open like your mailbox most probably has a lock. But it’s still lying outside.

Someone who is really mean could craft a key and go as far as discreetly stealing your mail for days. That’s what would happen to your users if someone manages to run a script injection attack in your application. They can craft a JS script that reads the localStorage, and sends them the tokens of any of your users affected by the breach!

There are ways to prevent cross-scripting attacks. It’s not a fatality, and some would argue that it’s still better than using an HTTP-only cookie! But if you go the web storage way, don’t forget to protect your mailbox.

JWTs Passed As Header: It’s A Pattern For API Calls Only

I’d like to go one step further and understand why so many developers are tempted to use the browser storage without even considering the cookie option.

I think there are two main reasons:

1. The API doesn’t use Set-Cookie during the login step. It often happens in APIs that were not specifically designed to be accessed from a browser, and they use an authentication pattern that works for servers, too (so no cookies). Your back-end developer will claim it was a “design choice.” Don’t trust them.
2. The client needs to set an Authorization header with the token. That’s just the symmetrical issue; the API chose not to rely on cookies.

You’ll meet a lot of documentation in the wild that shows headers like this:

Authorization: Bearer <token>

Setting the Authorization header implies you cannot use an HTTP-Only cookie. Of course, you can set the headers of a request using JavaScript. However, this means that the token must be available to JavaScript in the first place. Let’s repeat it once again: it’s ok not to use a cookie and prefer web storage, but be very careful with XSS attacks!

Moreover, this pattern works only for programmatic requests with fetch and XHR. You can’t secure access to the web page itself because the browser won’t set this header automatically during navigation. You just really can’t.

If you need to secure web pages, the backend of your website (which may differ from your actual API that serves data) should use HTTP-only cookies, or you should set a non-HTTP-only cookie manually. Either way, be careful that by using cookies, you are introducing a new vector of attacks.

Not all websites need to be secured this way, though. It’s actually pretty uncommon for public-facing websites. Remember Blitz.js’ motto “secure data, not pages.” But the cookie way is still a pattern you must be aware of.

If you really need to secure a web page, reach out to your backend developer and explain to them that cookies are good. They won’t contradict this claim.

If it’s really not possible to improve the API, you might want to adopt a pattern named “backend-for-frontend” (BFF). Basically, have a tiny server owned by the front-end team that takes care of calling authenticated APIs server-side, among other things. It works pretty well with frameworks like Remix or Next.js that have built-in server-oriented features.

Side Note On Basic Auth: Looks Like A Header, Behaves Like A Cookie

Of course, every rule must have its exceptions. There is only one common scenario I know where the token can be passed as a request header without using JavaScript: using basic authentication.

Basic authentication is an unsafe pattern, as identifiers are sent alongside every request, not just a token but the actual username and password. It’s a bit like asking your local drug dealer to pay your rent and give them your ID card and 2000$ that they should kindly bring to your landlord. It’s only ok if this is a fake ID. Actually, no, fake IDs and drug dealers are not ok. But you get the point!

It’s still interesting for a few use cases, like quickly securing the demo of a website. As long as you can accept the password to be stolen without that many consequences, typically if the password has been generated by an admin for a temporary occasion (NOT a user-set password that could be reused for multiple services!).

In basic authentication, the browser sets the Authorization header for you. You still need to set the credentials option properly when using fetch programmatically so that the header is also set in this case. But you don’t have to manually tweak the header. It really behaves like a cookie.

That’s actually one reason why this fetch option is named “credentials” and not just “cookies”: it also applies to the basic authentication header.

Conclusion

Wow, if you’ve read this far, you definitely earned your right to a quick summary!

But before that, I have another gift for you: an open-source demo of authentication with Deno and HTMX for the front end!

This is a very basic setup, but it puts the concepts described in this article into action using the “cookie way.”

Here is your well-deserved TL;DR:

• JWT is like banknotes, and sessionId is like credit cards. They are two authentication patterns, but either way, you want to store those tokens safely client-side.
• One safe place to store tokens in a browser is an HTTP-only and secure cookie, set via the Set-Cookie header of the login response. It cannot be stolen by malicious JavaScript code. Web storage is another possibility if you only secure API calls and not web page access. If you pick web storage, then be very careful with XSS attacks.
• Cookies are automatically sent alongside every request provided you use the right options, namely the path and SameSite attributes of the cookie. Like a wallet, you would carry it on you all the time and take it out of your bag only in safe places when a payment is needed.
• When navigating via URL, the browser takes care of sending the cookie containing the auth token (if properly set). But you may also need to call authenticated APIs using JavaScript code. Fetch credentials and XHR withCredentials options have to be configured correctly, so cookies are also sent alongside those programmatic requests. Don’t forget to set the Access-Control-Allow-Credentials response header as well.
• Auth is about politeness: the server must set the proper headers for everything to work. You can’t enter a bank and yell at people to get your money.
• This is particularly true for cross-origin requests. Companies must have secure and polite discussions with their bank, which can be separate organizations or located in a distant country. That’s what CORS is for websites and APIs. Don’t forget to properly set the Access-Control-Allow-Origin and Access-Control-Allow-Credentials headers in the response.
• If you store your savings in your mailbox, put a camera on top of it, and if you store your token in the web storage, protect it against XSS attacks.
• Banks don’t treat people like they treat other banks. Don’t confuse webpages patterns (have to rely on cookies and browser’s built-in features) and API patterns (relying on headers, which can be implemented using client-side JavaScript).
• Basic auth is a fun pattern that blurs the line, as it uses an Authorization header like for APIs, but it can be used for webpages too. It’s a specific and very insecure pattern that is only meant for a handful of limited use cases. Basic auth headers are a reason why fetch and XHR options are named “credentials” and not just “cookies.”

Resources

• Sending programmatic requests with fetch (as opposed to accessing a URL or submitting an HTML form and letting the browser send the request):
  • Using the Fetch API, Mozilla docs
  • fetch(), Mozilla docs
• Intro to JWT token.
  You’ll notice the ambiguity: it states that JWT should not be stored in browser storage, yet they should usually be passed via header… which implies storing the token in browser storage. But that’s not the only choice. HTTP-only cookies can be used too, and which is the best is an open debate.
• “Web Storage: the lesser evil for session tokens”, James Kettle
  HTTP-only cookies are debated: an article in favor of the web storage.
• Reflections on the REST architectural style and “principled design of the modern web architecture” (impact paper award), Fielding, R. T., Taylor, R. N., Erenkrantz, J. R., Gorlick, M. M., Whitehead, J., Khare, R., & Oreizy, P. (2017, August). In Proceedings of the 2017 11th joint meeting on foundations of software engineering (pp. 4-14).
  This paper from Fielding et al. states that the use case of session management has been overlooked in the past when defining the REST architectural style, leading to confusion and poor technological solutions
• You’ll find many great tutorials in the wild. A quick search for “authentication” on Smashing Magazine will confirm that there are already two great pieces just for Next.js:
  • “Dynamic Data-Fetching In An Authenticated Next.js App”, Caleb Olojo (Self-hosted authentication)
  • “How To Implement Authentication In Next.js With Auth0”, Facundo Giuliani
• Standard RFC for JWT
• “The Current State Of Authentication: We Have A Password Problem”, Drew Thomas
  There are other ways to authenticate the user than passwords. Most of the content of my article should, however, still hold even with other approaches (magic link, OpenID Connect…).
• The OPTIONS HTTP verb used by the CORS preflight request (triggered on non-simple requests like calling a JSON API with fetch)
• Breaking CORS preflight request to bypass CORS
• “Exploiting CORS misconfigurations for Bitcoins and bounties”, James Kettle
  Why the scheme (HTTP or HTTPS) matters with CORS; some exploit examples.
• The Wildcard exception
• Fetch standard definition of a “client”
• Excellent explanation on Access-Control-Allow-Origin, the header that controls CORS
• About CORS, and definition of “simple requests”
• XHR withCredentials in Axios
• CSRF
  A type of attack to bypass cross-origin restriction (not covered in this article, great read to go further, especially if you go “the cookie way” for API authentication)
• Limits of storing tokens in browser storage
• SameSite
  Configure the cookies to be sent or not when coming from another site
• About the SameSite attribute for cookies, from web.dev
• Subdomain takeover
• What is a “top-level navigation” in the browser
• “Protect your resources from web attacks with Fetch Metadata”, Lukas Weichselbaum
  Another brilliant piece from web.dev about Sec-Fetch headers.
• The canonical definition of Backend For Front-end.
  It’s similar to using a centralized API gateway for the front end, except that BFF is decentralized.
• “Cache your CORS, for performance & profit”, Tim Perry
  How to cache preflight requests to reduce your API workload (non-standard at the moment but extremely useful at scale).
• Redirecting in Next.js

(yk, il)

This article is sponsored by Flatfile

With the explosive growth of cloud computing over the last decade, unprecedented volumes of data — customer data, product data, statistics, financials, and so on — are being shared between organizations every day. While it would be great if there were a universal API that could guarantee secure and accurate transfer of data, the reality is much more primitive.

Most data that is being shared between companies these days is contained in CSV (comma-separated values) files. While CSVs are generally easy to create, they’re notoriously difficult to secure.

Because of this, the exchange of CSV files has the potential to cause serious problems for companies. And when it comes to user security and privacy, companies can’t afford to gamble on such liability.

How To Create A Secure Data Importer For Your Clients

TechRepublic recently published the findings from a KPMG report regarding data privacy. 64% of respondents said that they don’t believe that companies do much in the way of securing and protecting the data that’s been shared with them.

We know what the solution to this is and how to reduce those justifiable concerns. The first piece is to handle customer data responsibly and be transparent about what you’re doing with it. Regulations like GDPR and HIPAA provide the framework for this.

The other solution is to use technology that prioritizes user security. Just as you’d only add secure data handling features to your digital product — like contact forms, payment processors, and so on — the same applies to your data importer.

CSV importers are already a step in the right direction when it comes to security. Rather than sending email files back and forth over insecure email platforms, companies pass their data through CSV importers. The trick is to build or use a data importer that prioritizes security.

Next, you can find some things your importer will need in order for that to be true.

Protect Your Data With A Secure Infrastructure

When you build a website or app, there are certain measures you take to secure it. One of the most important measures is choosing a hosting provider with the proper infrastructure to support, stabilize and secure your digital product and the data that moves through it.

If you’re building your own data importer, then your product hosting will serve as the underlying infrastructure for it. Just make sure that it is capable of protecting the integrity of your product as well as securing the data transmissions that take place through your importer.

If you’re going to use a pre-built data importer solution, then spend some time reviewing the technology and systems that power it. Your users — and their customers — won’t be too happy if a data breach occurs and you try to put the blame on an external solution.

Here are some things that a secure data importer needs in terms of infrastructure:

Built In The Cloud

Cloud hosting offers a high degree of protection. When reviewing data importer options, take a look under the hood of each to confirm that they’re running in the cloud.

For instance, Flatfile’s servers are built on Amazon Web Services (AWS) cloud infrastructure. As a result, data that passes through Flatfile’s systems is fully encrypted using the AES-256 block cipher. This encryption protects data while it passes through the data importer as well as once it’s stored.

Security Testing And Monitoring

You and your clients aren’t the only ones who should be keeping an eye on what’s going on with your data importer. The company that devised the solution should be doing so, too.

There are a number of ways to ensure that the data importer and its infrastructure haven’t been compromised:

• Application monitoring;
• Continuous logging;
• User action tracing;
• Penetration testing;
• Malicious activity monitoring;
• Automated blocking.

It’s also important to find a data importer solution and provider that will be transparent about detected issues and alert you to any they’ve found.

Resource Management

Application performance goes hand-in-hand with security. This is critical for companies like EmployUS who promise users that their data will be secured, compliant and available.

In addition to reviewing your data importer solution for security features, also look for what it’s doing to optimize performance and uptime.

Load balancing and resource scaling are two things to look for. Another thing you should do is check out the company’s “Status” page. Here’s an example of what the “Status” page looks like for Flatfile:

If there are issues with any aspect of the data importer technology, you’ll find proof of it on this page. Users can also subscribe for real-time updates. Having this level of visibility and transparency is essential when you outsource a critical piece of your application to another provider’s solution.

Ensure Regulatory And Legal Compliance

Different types of digital products have to maintain certain levels of compliance. This can be due to the types of data they handle (like in the medical and financial industries) or because of where they or their users are located in the world.

Whether you’re building your own importer or using a pre-built solution, your technology and data handling processes need to be compliant with all relevant security and privacy regulations.

For example, Flatfile’s solution maintains compliance with the following:

• GDPR
  Although this data security and privacy regulation was passed to protect EU citizens’ personal data, it has far-reaching effects. Because many businesses these days serve customers all around the world, GDPR compliance is essential for anyone doing business online.
• AICPA SOC 2 (Types I and II)
  The Association of International Certified Professional Accountants has its own regulations related to data privacy and protection. SOC and SOC 2 refer to the audit that service providers must pass in order to ensure they’re securely handling employee and customer data.
• EU/U.S. Privacy Shield
  The U.S. Department of Commerce put together this framework in conjunction with the European Commission and the Swiss Administration. It provides companies that conduct transatlantic commerce with a set of data protection requirements to follow when transferring data.
• HIPAA
  HIPAA is a U.S. law concerned with the protection of sensitive patient data. It ensures that their health information is private and secure. It also gives patients more control over how their information is used and to whom it is disclosed.

With so many regulations to stay on top of, a data importer can become a huge chore to maintain and update. This is why many developers and companies choose to use a pre-built data importer solution.

Osmind, for instance, not only streamlined its data transfer process with Flatfile Workspaces, but it enabled them to achieve HIPAA compliance — something that’s critical when working with sensitive health records.

Bottom line: By finding a data importer that maintains various regulatory compliances, you won’t have to spend time down the road looking for alternative solutions to fill in the missing gaps. Plus, a provider that keeps its systems updated as regulations and standards change will greatly reduce the risk of your data importer falling out of compliance.

Prevent Your Importer From Breaking So Easily

Whether you are populating databases for a warehouse catalog, an ERP, or just a list of every town in which you operate, your importer needs to be strong.

For instance, let’s say a user ignores your file preparation instructions and rushes to import the files they have. Before it even gets to the point of cleaning up the data, you want to make sure the importer is able to process it without breaking down.

A broken data importer can leave users with a bad impression of the product they’re using and the company behind it. It doesn’t matter if it’s their fault for not reading the instructions or for poorly formatting their file. Encountering a broken feature is frustrating and can quickly lead to concerns with regard to security and privacy.

“What happened?”
“Did my data even go through?”
“Should I try it again, or is it too risky?”

With how advanced technology has gotten today, users will likely wonder why you hadn’t anticipated these kinds of issues and sorted them out already. So, in order to prevent end users from encountering a broken data importer, it will need to be smart and flexible.

This means using a data importer that:

• Provides no more than a few guidelines so that users don’t have to read an entire manual in order to prepare their files;
• Moves massive amounts of files with thousands of rows of data without erroring out;
• Accepts files just as the customer has prepared them;
• Easily maps and validates data no matter how inconsistent or varied the formats are;
• Detects and notifies you (or your users) of serious errors before uploading.

An importer that breaks down all the time is going to cause issues for everyone involved. So too will one that brings tons of garbled data into your system — especially when that data is mission-critical.

By creating or using a strong and agile data importer, you can reduce the frequency with which errors occur. This will make your data importer more reliable and valuable to your users and help them instill greater trust in their own customers.

Wrapping Up

User security — as well as the perception of how secure the products are that they use — should matter a good deal to companies who collect data from their customers. That’s why it’s essential for developers to use CSV importers that they trust and that won’t put their clients or their end users in harm’s way.

As for whether you should build or buy a data importer, that decision is yours to make. However, if security and compliance are top priorities, then purchasing a pre-built importer like Flatfile would be the more economical and practical choice.

(yk, il)

Q. Did any part of your education require you to read the building regulations?
A. No, I don’t recall, but I don’t think so.
Q. What about fire safety of building materials?
A. No.

— Witness testimony, the Grenfell Tower inquiry, day 22

As the hard copies of Understanding Privacy begin to ship all over the world, the brilliant team at Smashing, who helped me bring my book to life, have asked me to share a few thoughts on what it might achieve in the months and years to come.

I wrote the book for two broad audiences, and if you’re reading this, you probably fall into one of them. The first audience is designers, developers, and project managers already working on the open web, either professionally or in side projects. The second audience is students and future professionals in those fields, whether they are in secondary schools, undergraduate courses, vocational training, or code academies.

The reason I structured the book in the way I did is that these two audiences, and you, tend to have something in common: you’ve never received any previous training or education on positive foundational privacy, either as a concept, a legal issue, or a professional practice, either in your formal education (assuming you had any) or in your workplaces.

What you have been told about privacy, by contrast, has been defined by a lot of high-level legalese jargon, compliance scaremongering, and terrifying headlines about the surveillance society that exists around us — whether we know it or not.

“

That means that an entire generation of professionals like you have been introduced to privacy by being thrown into advanced legal compliance headaches or reactive fixes to the problems created by others, with no knowledge of the basic concepts and principles about what privacy actually is and how to achieve it.

I hope that Understanding Privacy can go some way towards giving people like you a confident understanding of those foundational concepts. Indeed, it’s my hope that teachers and educators will use it as the basis for a curriculum on privacy so that a healthy approach to privacy becomes baked-in from the start rather than retrofitted at the end.

That, of course, raises two further issues.

1. The first is how teachers and educators find the book in the first place;
2. The second is how the lessons in it reach developers (both current and future ones) who have never had a teacher and never will.

The question of how we teach privacy, what we teach, and where we teach it, has troubled me for years. That’s a hard enough problem to crack. After all, web development is an unorganized field. In fact, in the strictly legal sense, it is not a profession.

What do I mean by that? Professions are defined by industry-based organisations, common paths of entry, common educational requirements, continuing professional development, and even certifications that require refresher training every few years. Being a professional, in the strictly legal sense, means that there is a body made up of your peers who make sure that you bring a common body of knowledge to the work you put into the world and that the work you do meets externally verifiable standards.

Web development, on the other hand, is an unorganized and unstructured field that anyone can enter, at any time, with any form of formal training or with none at all, and without any external certifications or approval. A software engineer who went through a four-year bachelor’s degree program in computer science can be working on the same team, doing the same work, as a former airline pilot who learned code for fun. It’s that occupational diversity that has contributed to the growth of the open web as a whole; indeed, I find that the best teams contain people who approach their work from the diverging perspectives they gained doing something completely different.

But it also means that the knowledge of privacy that we bring to our work is, quite simply, all over the place. If that knowledge is there at all. And without a common pathway of education, whether that’s access to training and continuing professional development or work towards a standard of foundational knowledge, we will continue to bring the contrasting social, legal, and cultural differences to privacy which I discuss in Part One to the work we put into the world.

And our users will continue to pay the price for that.

As I write in the book, we can’t wait for educators, employers, and institutions to fill the gaps in our knowledge. Educating ourselves on privacy has never been more important. It pains me to know that the book is the first and only education on privacy that most of its readers will ever have, but something is better than nothing. Unfortunately, it won’t be enough.

Because the dilemma of how we teach a positive foundational approach to privacy — in an unorganized and open industry — has taken on a whole new urgency through my pivoted career into the politics of tech. And that situation is far scarier than you can imagine.

Much of my time in recent years has been spent retorting various regulatory plans for personal liability regimes in digital regulation. That means that politicians increasingly want to hold the people who make the open web legally and even criminally responsible for any misuse or unintended consequences of their work.

Some of this is born out of pressure to “do something” about the mess that the open web is today; sometimes it’s about “reining in the tech giants” (and I can tell you that politicians absolutely think Facebook is the Internet); sometimes it’s about barefaced moves for political power (hello from Brexit Britain); and sometimes it’s about cracking down on public discourse and interaction, delegating the requirement for censorship and control to the tech sector and therefore to workers like you.

Whatever reasons are behind these proposals, they are not going away. In fact, they’re only getting louder.

Many of these proposed liability regulations have been borrowed from traditional health and safety regimes. But these draft regulations, and those who support them, fail to understand that human discourse cannot be regulated as if it was fire-retardant cladding on a building that wasn’t fire-retardant at all (as I noted in the quote which began this article). By trying to shoehorn human interactions into a “risk assessment” model, these regimes risk creating an unworkable legal standard where a person who misuses a service is not deemed liable, but the person who built the service is.

These proposed liability regimes, for what it’s worth, have been drafted in a highly obsessive and vindictive manner to target a handful of high-profile American billionaires and celebrities in a handful of American big tech companies. (To be precise, these proposed regimes target three specific individuals in two companies, as if their arrests and imprisonment would fix all the problems on the Internet.) For the purposes of this discussion, those men’s guilt or complacency is neither here nor there.

That’s because, for a range of obvious reasons, once those laws are on the books, the celebrity billionaires will be able to afford to duck and dodge the charges. But because politicians insist that “something has to be done” and “someone needs to pay for this”, those laws will be used, instead, to go after the little guys and the easy targets. That means you.

What I am saying is that policymakers across all societies and cultures are turning their attention to people like you, the knowledge you bring to the table, and the work you put into the world. They’re not doing that because they want to support you into the next phase of your career. They’re doing that because they’re looking for someone to blame. They need someone to blame.

In fact, I have encountered politicians who are desperate to actually arrest, prosecute, and imprison developers, hopefully in front of the TV cameras, as punishment for the sins of their celebrity bosses. Those policymakers are in ascendancy, and they are not going away.

And when they’re looking for someone to blame for the problems on the open web today and need an easy target to take down for a quick political “win,” there you are, with no qualifications or foundational training or formal education, making things that millions of people use.

I think you see where this is going.

I wrote Understanding Privacy to contribute to a better open web, and I wrote it in the most positive and constructive tone possible (and hey, that was hard going in lockdown). But I would not be serving the people I wrote it for if I pretended that the book’s teachings exist in a happy bubble where the fixes are easy. The book’s teachings exist in a political climate where the people who make the web, including you, are now a target.

I want the book to contribute to a better standard of privacy for the people we build the web for. By reading the book, you’ll learn how to protect them in everything you do, regardless of the presence or absence of any privacy legislation. But in the political climate that exists around us all, by reading the book, you’ll learn how to protect yourself too.

In the absence of any formal curricular and educational path, workplace training, professional body, or legal standard, the book will help you to create an accountable and documented framework around privacy in your work — no matter who employs you or what you’re working on. That framework, and for that matter, the book, can’t protect you on its own. But the lessons you learn from it might just help you when the day comes that it’s you and your team in your co-working lounge, and not the celebrity billionaires and their teams in Silicon Valley, who become the target for an ambitious politician’s campaigning.

During life in lockdown, we all became familiar with the “oxygen mask” rule: secure your own mask before putting one onto someone else. In other words, you can’t support others if you’re not supporting yourself. As you use Understanding Privacy to build a better web for your users, take some time to think about the ways you can use its lessons to protect yourself, especially in light of policymakers’ obsession with getting ad-hominem revenge on Big Tech celebrities — an obsession which views you as expendable collateral damage. And as I write in Part Four, think about the developers who will come after you and what sort of world they can build if they are given a better education in foundational privacy.

Or, at the very least, given more than just one book.

Details About The Book

• Quality hardcover, stitched binding, ribbon page marker.
• Free worldwide shipping from Germany.
• eBook is available as PDF, ePUB, and Amazon Kindle.
• ISBN: 978-3-945749-64-7 (print)
• Get the book (Print Hardcover + eBook)

(as, vf, il)

What is privacy? What exactly does it mean? How do we consider, manage and maintain privacy? And how dow do we design and build experiences that have privacy at their heart? That’s exactly what Understanding Privacy is all about: a practical guide to privacy on the web, from data collection and use of personal data to creating safe, inclusive experiences for everyone.

From the still-relevant Privacy by Design Framework to the recent conflicts over consent and health data, Heather Burns has created an indespensible resource for anyone working to build safety and trust into their interfaces.

• eBook is available as PDF, ePUB, and Amazon Kindle.
• Free worldwide shipping from Germany.
• Download a free PDF sample (11MB)
• Get the book (Print Hardcover + eBook)

About The Book

Understanding Privacy is a practical guide to the concepts and ideas that inform privacy on the web. It’s about all the fundamental values of privacy as a concept, which precede privacy as a legal compliance issue. It’s about the ways these concepts impact your work as a designer, a developer, or a project manager. And it’s about the ways you can adopt these principles to create a healthy, user-centric approach to privacy in everything you do.

Heather Burns, a tech policy and regulation specialist, explains what she has experienced working on privacy from every angle — human rights, law, policy, and web development — in the simplest way possible, and in the most positive way possible, in ways you can understand, use, and adapt in your work on the web right away.

This book is not a legal reference manual. After reading it, you will have shifted your understanding from a negative view of privacy as a scary legal compliance obligation to a positive view of privacy as an opportunity to build and design a better web. Download a free PDF sample (11MB).

288 pages. Written by Heather Burns. Cover design by Espen Brunborg. Both eBook and print shipping are now available.

You’ll Learn:

• Fundamental concepts, definitions and frameworks behind privacy and data protection,
• Healthy approach to user privacy into everything you build and design,
• Common privacy issues and how you can make a difference,
• How to lay the ground for future developers, designers, and project managers to build a better web for tomorrow,
• The obligations we have to safeguard user privacy and health data.

Who Is This Book For?

Understanding Privacy is for designers, developers, and project managers who want to understand what privacy really is about and who want to integrate a healthy approach to user privacy into everything they do not only to put their users first today but also to help build a better web for tomorrow.

Table Of Contents

288 pages. Both eBook and print shipping are now available. Written by Heather Burns. Cover design by Espen Brunborg.

About The Author

Heather Burns (@WebDevLaw) is a tech policy professional and an advocate for an open Internet which upholds the human rights to privacy, accessibility, and freedom of expression. She’s been passionate about privacy since she built her first web site in 1996, and has educated thousands of professionals worldwide on the fundamentals of a healthy approach to protecting people and their data. She lives in Glasgow, Scotland.

Reviews And Testimonials

“Heather's broad knowledge, experience, and ability to articulate these complex matters is nothing short of astounding. I’ve learned an amazing amount from her. She always informs and entertains, and she does so from the heart.”

Mike Little, Co-Founder of WordPress

“No more excuses for overlooking privacy: Heather’s guide is an essential toolbox for user-centric product developers and for anyone interested in building a better web. Expect the full sweep, from historical context and core concepts in US and EU privacy practice, to practical tips and advice — dispensed in highly readable style.”

Natasha Lomas, Senior Reporter, Techcrunch.com

“Privacy is an oft-talked about and rarely understood part of our modern digital lives. Heather has been on the forefront for the battle of our privacy for decades. In this book she makes the case for why privacy is one of the foundational pillars on which our society rests, and why eroding our privacy means eroding a cornerstone of our lives, our communities, and our democracy. A must-read for anyone working on or with the web.”

Morten Rand-Hendriksen, Senior Staff Instructor, LinkedIn Learning

“Privacy can seem complicated but it doesn’t need to be. Heather covers all that you need to know with astonishing clarity. This book gives you all you need to understand and handle privacy work, and makes for great teaching material that experts could rely on.”

Robin Berjon, former Head of Data Governance at The New York Times

Technical Details

• ISBN: 978-3-945749-64-7 (print)
• Quality hardcover, stitched binding, ribbon page marker.
• Free worldwide shipping from Germany.
• eBook is available as PDF, ePUB, and Amazon Kindle.
• Get the book (Print Hardcover + eBook)

Community Matters ❤️

Producing a book takes quite a bit of time, and we couldn’t pull it off without the support of our wonderful community. A huge shout-out to Smashing Members for the kind, ongoing support. The eBook is and always will be free for Smashing Members. Plus, Members get a friendly discount when purchasing their printed copy. Just sayin’! ;-)

More Smashing Books & Goodies

Promoting best practices and providing you with practical tips to master your daily coding and design challenges has always been (and will be) at the core of everything we do at Smashing.

In the past few years, we were very lucky to have worked together with some talented, caring people from the web community to publish their wealth of experience as printed books that stand the test of time. Steven and Stefan are two of these people. Have you checked out their books already?

(as, il)

To many of us, privacy might feel like a complex, abstract concept. We can’t hold privacy in our hands, we can’t touch it, we can’t explore its volume or shape with our eyes or our fingertips. Surely it’s a part of each of us, yet it feels so intangible and so invisible — beyond reach and out of view.

So what is privacy? What exactly does it mean? How do we consider, manage and maintain privacy? And how dow do we design and build experiences that have privacy at their heart? That’s exactly what Understanding Privacy is all about: a practical guide to privacy on the web, from data collection and use of personal data to creating safe, inclusive experiences for everyone. Jump to table of contents ↓

• eBook now available.
• Printed books start shipping in 2022.
• Download a free PDF sample (11MB)
• Get the book (Print Hardcover + eBook)

About The Book

Understanding Privacy is a practical guide to the concepts and ideas that inform privacy on the web. It’s about all the fundamental values of privacy as a concept, which precede privacy as a legal compliance issue. It’s about the ways these concepts impact your work as a designer, a developer, or a project manager. And it’s about the ways you can adopt these principles to create a healthy, user-centric approach to privacy in everything you do.

Heather Burns, a tech policy and regulation specialist, explains what she has experienced working on privacy from every angle — human rights, law, policy, and web development — in the simplest way possible, and in the most positive way possible, in ways you can understand, use, and adapt in your work on the web right away.

This book is not a legal reference manual. After reading it, you will have shifted your understanding from a negative view of privacy as a scary legal compliance obligation to a positive view of privacy as an opportunity to build and design a better web. Download a free PDF sample (11MB).

288 pages. Written by Heather Burns. Cover design by Espen Brunborg. eBook now available, print shipping in early December.

You’ll Learn:

• Fundamental concepts, definitions and frameworks behind privacy and data protection,
• Healthy approach to user privacy into everything you build and design,
• Common privacy issues and how you can make a difference,
• How to lay the ground for future developers, designers, and project managers to build a better web for tomorrow,
• The obligations we have to safeguard user privacy and health data.

Who Is This Book For?

Understanding Privacy is for designers, developers, and project managers who want to understand what privacy really is about and who want to integrate a healthy approach to user privacy into everything they do not only to put their users first today but also to help build a better web for tomorrow.

Table Of Contents

288 pages. eBook now available, print shipping in early December. Written by Heather Burns. Cover design by Espen Brunborg.

About the Author

Heather Burns (@WebDevLaw) is a tech policy professional and an advocate for an open Internet which upholds the human rights to privacy, accessibility, and freedom of expression. She’s been passionate about privacy since she built her first web site in 1996, and has educated thousands of professionals worldwide on the fundamentals of a healthy approach to protecting people and their data. She lives in Glasgow, Scotland.

Reviews and Testimonials

“Heather's broad knowledge, experience, and ability to articulate these complex matters is nothing short of astounding. I’ve learned an amazing amount from her. She always informs and entertains, and she does so from the heart.”

Mike Little, Co-Founder of WordPress

“No more excuses for overlooking privacy: Heather’s guide is an essential toolbox for user-centric product developers and for anyone interested in building a better web. Expect the full sweep, from historical context and core concepts in US and EU privacy practice, to practical tips and advice — dispensed in highly readable style.”

Natasha Lomas, Senior Reporter, Techcrunch.com

“Privacy is an oft-talked about and rarely understood part of our modern digital lives. Heather has been on the forefront for the battle of our privacy for decades. In this book she makes the case for why privacy is one of the foundational pillars on which our society rests, and why eroding our privacy means eroding a cornerstone of our lives, our communities, and our democracy. A must-read for anyone working on or with the web.”

Morten Rand-Hendriksen, Senior Staff Instructor, LinkedIn Learning

“Privacy can seem complicated but it doesn’t need to be. Heather covers all that you need to know with astonishing clarity. This book gives you all you need to understand and handle privacy work, and makes for great teaching material that experts could rely on.”

Robin Berjon, former Head of Data Governance at The New York Times

Technical Details

• ISBN: 978-3-945749-64-7 (print)
• Quality hardcover, stitched binding, ribbon page marker.
• Free worldwide shipping from Germany, starting in early December 2022.
• eBook is already available as PDF, ePUB, and Amazon Kindle.
• Get the book (Print Hardcover + eBook)

Community Matters ❤️

Producing a book takes quite a bit of time, and we couldn’t pull it off without the support of our wonderful community. A huge shout-out to Smashing Members for the kind, ongoing support. The eBook is and always will be free for Smashing Members. Plus, Members get a friendly discount when purchasing their printed copy. Just sayin’! ;-)

More Smashing Books & Goodies

Promoting best practices and providing you with practical tips to master your daily coding and design challenges has always been (and will be) at the core of everything we do at Smashing.

In the past few years, we were very lucky to have worked together with some talented, caring people from the web community to publish their wealth of experience as printed books that stand the test of time. Steven and Stefan are two of these people. Have you checked out their books already?

Privacy by design is possible, but over time, it has become challenging to build and deploy sites and apps that use personal information in practical ways that are safe and ethical. In fact, many of us have given up at least a little of our own privacy online.

And so in our work, it may become easier to rationalize collecting more user data than we need, to avoid transparency around how that data is used, and even to use that data for unintended purposes. We tell ourselves it is for the convenience of our users, or to enhance their experiences. It’s no secret that unethical data practices can cause harm, often in ways we can’t predict.

Thoughtful data collection, storage, and use of data is possible, though. Heather Burns brings clarity to the subject by explaining the principles behind the collection, storage, and use of personal data, and how to use those principles to create safer experiences for your users. Understanding Privacy is an essential book for anyone that collects personal information. Jump to table of contents ↓.

• eBook available for download in October 2022,
• Printed books will be shipped in early December 2022.
• Pre-order the book.

About The Book

Understanding Privacy is an introduction to the beliefs, concepts, and ideas that inform privacy as it exists — or has failed to exist — on the open web that we build. It’s about all the fundamental values of privacy as a concept, which precede privacy as a legal compliance issue. It’s about the ways these concepts impact your work as a designer, a developer, or a project manager. And it’s about the ways you can adopt these principles to create a healthy, user-centric approach to privacy in everything you do.

Heather explains what she has experienced working on privacy from every angle — human rights, law, policy, and web development — in the simplest way possible, and in the most positive way possible, in ways you can comprehend, use, and adapt in your work on the web right away.

A healthy approach to user privacy doesn’t tell you how to code. It tells you how to make the right decisions which inform the code. It also gives you the foundation you need to question, and even challenge, workplace practices which might not be in your users’ best interests.

This book is not a legal reference manual. By the end of this book, you will have shifted your understanding from a negative view of privacy as a scary legal compliance obligation to a positive view of privacy as an opportunity to build a better web.

Approx. 350 pages. Written by Heather Burns. Cover design by Espen Brunborg. eBook available for download in October, print shipping in early December.

You’ll learn:

• Fundamental concepts, definitions, and frameworks behind privacy and data protection,
• how to integrate a healthy approach to user privacy into everything you do,
• common privacy issues and how you can make a difference,
• how to lay the ground for future developers, designers, and project managers to build a better web for tomorrow,
• the obligations we have to safeguard user privacy and health data.

Who is this book for?

Understanding Privacy is for designers, developers, and project managers who want to understand what privacy really is about and who want to integrate a healthy approach to user privacy into everything they do — not only to put their users first today but also to help build a better web for tomorrow.

Table Of Contents

Approx. 350 pages. eBook available for download in October, print shipping in early December. Written by Heather Burns. Cover design by Espen Brunborg.

About the Author

Heather Burns is a tech policy and regulation specialist. She advocates for an open web built around international standards of human rights, privacy, accessibility, and freedom of expression. She’s currently also an Internet Society Mid-Career Fellow.

Technical Details

• ISBN: 978-3-945749-64-7 (print)
• Quality hardcover, stitched binding, ribbon page marker.
• Free worldwide airmail shipping from Germany starting in early December 2022.
• eBook available for download in October 2022 as PDF, ePUB, and Amazon Kindle.
• Pre-order the book.

Community Matters ❤️

Producing a book takes quite a bit of time, and we couldn’t pull it off without the support of our wonderful community. A huge shout-out to Smashing Members for the kind, ongoing support. The eBook is and always will be free for Smashing Members as soon as it’s out. Plus, Members get a friendly discount when purchasing their printed copy. Just sayin’! ;-)

More Smashing Books & Goodies

Promoting best practices and providing you with practical tips to master your daily coding and design challenges has always been (and will be) at the core of everything we do at Smashing.

In the past few years, we were very lucky to have worked together with some talented, caring people from the web community to publish their wealth of experience as printed books that stand the test of time. Steven and Stefan are two of these people. Have you checked out their books already?

A newspaper sign-up form had fields for name, email, and password. So, I started typing on the name field, and the autofill suggested my profile. But there was something funky. The autocomplete suggestion included my mailing address. Needless to say, it was puzzling: the address was not a field in the form. Why was it even suggested?

By the time this question started forming in my mind, my brain had already signaled my finger to click on the suggestion, and it was done. Next, I was taken to a second form page which requested additional information like address, phone, date of birth, and so on. And all those fields had been pre-populated by the autofill functionality, too.

I sighed in relief. It was a “multi-stepped” form and not a trick by the website. It was a reputable newspaper, after all. I deleted all the optional information from the second page, completed the sign-up, and moved on.

That (troubled) interaction highlighted one of the risks of using autocompletion features.

Autocomplete And Autofill

They may sound similar, but autocomplete and autofill are not the same thing. Although they are closely related:

• Autofill is a browser feature that allows people to save information (on the browser or the OS) and use it on web forms.
• autocomplete is an HTML attribute that provides guidelines to the browser on how to (or not to) autofill in fields in a web form.

We could say that autofill is the “what,” while autocomplete the “how”, i.e. autofill stores data and tries to match it in a web form (based on the fields’ name, type, or id), and autocomplete guides the browser on how to do it (what information is expected in each field).

Autocomplete is a powerful feature with many options that allows specifying many different types of values:

• Personal: Name, address, phone, date of birth;
• Financial: credit card number, name, expiration date;
• Demographics: location, age, sex, language;
• Professional: company and job title.

Autofill is a widespread feature either by choice or by accident: who hasn’t accepted to let the browser save/use web form information, either on purpose or by mistake? And that could be a problem — especially combined with bad use of autocomplete (and the added outrageous number of phishing emails and SMS messages nowadays.)

Privacy Risks

Both of these features present (at least) two main risks for the user, both related to their personal data and its privacy:

1. Non-visible fields are populated (this is not the same as fields with a hidden type);
2. Autocompleted information can be read via JavaScript even before the user submits the form.

This means that once a user selects to autofill the information, all the fields will be available for the developer to read. Again, independently of the user submitting the form or not, without the user knowing what fields were actually populated.

This last part is relative: knowing what fields are populated will depend on the browser. Safari and Firefox do a good job at this (as we’ll soon see below). On the other hand, Chrome, the most popular browser at the moment, offers a bad experience that may trick even the most knowledgeable users into sharing their personal information.

If we also consider the times in which the user accidentally chooses to populate the fields, this issue becomes more relevant. Let’s check it in more detail with an example.

A Little Experiment

I ran a little experiment creating a form with many fields and attaching the autocomplete attribute with different values. Then, I played a little with the form’s structure:

• I hid most of the fields by putting them in a container offscreen (instead of using hidden or type="hidden");
• I removed the visually hidden fields from the tab order (so keyboard users would overlook the hidden fields);
• I tried sorting the fields in a different order (and to my surprise, this impacted the autofill!).

In the end, the code for the form looked like this:

<form method="post" action="javascript:alertData()">
  <label for="name">Full name</label><input id="name" name="name" autocomplete="name" /><br/>
  <label for="email">Email</label><input id="email" name="email"/><br/>
  <label for="postal-code">ZIP</label><input id="postal-code" name="postal-code" autocomplete="postal-code"/>
  <div style="position:absolute;top:-10000in" class="hide-this">
    <!-- Hidden -->
    <label for="firstname">First name</label><input tabindex="-1" type="hidden" id="firstname" name="firstname" autocomplete="given-name" /><br/>
    <label for="lastname">Last name</label><input tabindex="-1"  id="lastname" name="lastname" autocomplete="family-name" /><br/>

    <label for="honorific-prefix">honorific-prefix</label><input tabindex="-1" id="honorific-prefix" name="honorific-prefix" autocomplete="honorific-prefix"/><br/>
    <label for="organization">Organization</label><input tabindex="-1" id="organization" name="organization" /><br/>
    <label for="phone">Phone</label><input tabindex="-1" id="phone" name="phone" autocomplete="tel" /><br/>

    <label for="address">address</label><input tabindex="-1" id="address" name="address" autocomplete="street-address" /><br/>
    <label for="city">City</label><input tabindex="-1" id="city" name="city" autocomplete="address-level2" /><br/>
    <label for="state">State</label><input tabindex="-1" id="state" name="state" autocomplete="address-level1" /><br/>
    <label for="level3">Level3</label><input tabindex="-1" id="level3" name="state" autocomplete="address-level3" /><br/>
    <label for="level4">Level4</label><input tabindex="-1" id="level4" name="state" autocomplete="address-level4" /><br/>
    <label for="country">Country</label><input tabindex="-1" id="country" name="country" autocomplete="country" /><br/>

    <label for="birthday">Birthday</label><input tabindex="-1" id="birthday" name="birthday" autocomplete="bday" /><br/>
    <label for="language">Language</label><input tabindex="-1" id="language" name="language" autocomplete="language" /><br/>
    <label for="sex">Sex</label><input tabindex="-1" id="sex" name="sex" autocomplete="sex" /><br/>
    <label for="url">URL</label><input tabindex="-1" id="url" name="url" autocomplete="url" /><br/>
    <label for="photo">Photo</label><input tabindex="-1" id="photo" name="photo" autocomplete="photo" /><br/>
    <label for="impp">IMPP</label><input tabindex="-1" id="impp" name="impp" autocomplete="impp" /><br/>

    <label for="username">Username</label><input tabindex="-1" id="username" name="username" autocomplete="username" /><br/>
    <label for="password">Password</label><input tabindex="-1" id="password" name="password" autocomplete="password" /><br/>
    <label for="new-password">Password New</label><input tabindex="-1" id="new-password" name="new-password" autocomplete="new-password" /><br/>
    <label for="current-password">Password Current</label><input tabindex="-1" id="current-password" name="current-password" autocomplete="current-password" /><br/>

    <label for="cc">CC#</label><input tabindex="-1" id="cc" name="cc" autocomplete="cc-number" /><br/>
    <label for="cc-name">CC Name</label><input tabindex="-1" id="cc-name" name="cc-name" autocomplete="cc-name" /><br/>
    <label for="cc-expiration">CC expiration</label><input tabindex="-1" id="cc-expiration" name="cc-expiration" autocomplete="cc-expiration" /><br/>
    <label for="cc-zipcode">CC Zipcode</label><input tabindex="-1" id="cc-zipcode" name="cc-zipcode" autocomplete="cc-postalcode" /><br/>
  </div>
  <button>Submit</button>
</form>

Note: I created this demo a while back, and the standard is a living document. Since then, some of the autocomplete names have changed. For example, now we can specify new-password and current-password or more details for address or credit card that were not available before.

That form had three visible fields (name, email and zipcode). While that form is common among insurance companies, cable, and other service providers, it may not be too widespread, so I reduced the form even more with a single email field. We see that everywhere to sign up to websites, newsletters, or updates. You can see a running demo here:

If you used autocomplete to fill in the form, you already shared more information than what you wanted (don’t worry, it’s all local and not shared with me). And in Chrome, it might have even looked like a perfectly normal subscription form.

If you don’t have/use autofill, don’t worry. Here’s a summary of how the experience is on three different browsers.

Note: All of these tests assume the use of Autofill and are based on a fake profile!

Safari

When you click on a form control, Safari will show an icon on the right side of the field. Clicking on it will show a pop-up with the information that the browser will share with the form:

A good thing: it displays all the data that will be shared as part of the form. Not only the data for the visible fields but all of them. At this point, the user may suspect something is not quite alright. There’s something fishy.

When I reduced the form to just the email field, Safari did something interesting. The autofill popup was different:

It states that it will only share the email (and it only does share that piece of information). But the contact info below may be trickier. When we click on that button, the browser shows a summary of the profile with its shared data. But that is not clearly stated anywhere. It simply looks like a regular contact card with some “share/do not share” options. After clicking on the “Autofill” button, the form is populated with all the data. Not only the email:

So there is a way for a user to share information with the form inadvertently. It’s tricky but not too far-fetched considering that it is the one “highlighted” with an icon out of the two possible options.

Funny thing, browsers separate the personal data from the credit card data, but Safari populated part of the credit card information based on the personal data (name and ZIP.)

Firefox

Using the autofill in Firefox is a bit more complex. It is not automatic like in Chrome, and there’s no icon like in Safari. Users will have to start typing or click a second time to see the autofill popup, which will have a note with every category that the browser will fill in, not only the visible fields:

Testing with the email-only form, Firefox presented the same autofill popup stating which fields categories it would populate. No difference whatsoever.

And similarly to the other browsers, after the autofill ran, we could read all the values with JavaScript.

Firefox was the best of the three: it clearly stated what information would be shared with the form independently of the fields or their order. And it hid the autofill functionality a second user interaction happened.

A keyboard user could select the autofill without realizing, by getting inside the popup bubble and pressing the tab key.

Chrome

Then it came the turn for Chrome. (Here I use “Chrome,” but the results were similar for several Chromium-based browsers tested.) I clicked on the field and, without any further interaction, the autofill popup showed. While Firefox and Safari had many things in common, Chrome is entirely different: it only shows two values, and both are visible.

This display was by design. I picked the order of the fields on purpose to get that particular combination of visible controls and autocomplete suggestions. However, it looks like Chrome gives some autocomplete properties more “weight” for the second value. And that makes the popup change depending on the order of the fields in the form.

Testing with the second version of the form was not much better:

While the popup shows a field that is not visible (the name), it is unclear what the purpose of the name is on the popup. An experienced user may know this happens because the name is shared, but an average user (and even the experienced ones) may think the email is associated with the profile with that name. There is zero indication of the data that the browser will share with the form.

And as soon as the user clicks on the autofill button, the data is available for the developer to read with JavaScript:

Chrome was the worst offender: it shared the information automatically, it was unclear what data was involved, and the autofill suggestions changed based on the controls’ order and attributes.

The first two issues are common to all/many browsers, to the point that it may even be considered a feature. However, the third issue is exclusive to Chromium browsers, and it facilitates a sketchy dark pattern.

This behavior would be more of an anecdote and not a problem if it wasn’t because Chrome takes a considerable market share of the browsers online (includes Chrome and Chromium-based).

The Dark Pattern

As you probably know, a dark pattern is a deceptive UX pattern that tricks users into doing things they may not really want to do.

“When you use websites and apps, you don’t read every word on every page — you skim read and make assumptions. If a company wants to trick you into doing something, they can take advantage of this by making a page look like it is saying one thing when it is in fact saying another.”

— Harry Brignull, darkpatterns.org

The behavior described in the previous points is clearly a deceptive user experience. Non-experienced users will not realize that they are sharing their personal data. Even more tech-savvy people may get tricked by it as Chrome makes it look like the selected option belongs to a profile instead of clearly stating what information is being shared.

The browser implementations cause this behavior, but it requires the developer to set it in place to exploit it. Unfortunately, there already are companies willing to exploit it, selling it as a feature to get leads.

As long as a dark pattern goes, it may also be an illegal one. This is because it breaks many principles relating to the processing of personal data specified in article 5 of the European General Data Protection Regulation (GDPR):

• Lawfulness, fairness, and transparency
  The process is all but transparent.
• Purpose limitation
  The data is processed in a way incompatible with the initial purpose.
• Data minimization
  It is quite the opposite. Data maximization: get as much information as possible.

For example, if you want to sign up for a newsletter or request information about a product, and you provide your email, the website has no legal right to get your name, address, date of birth, phone number, or anything else without your consent or knowledge. Even if you considered that the user gave permission when clicking on the autofill, the purpose of the obtained data does not match the original intent of the form.

Possible Solutions

To avoid the problem, all actors need to contribute and help fix the issue:

1. Users
2. Developers and designers
3. Browsers

1. Users

The only thing on the user side would be to ensure that the data displayed in the autofill popup is correct.

But we need to remember that the user is the victim here. We could blame them for not paying enough attention when clicking on the autofill, but that would be unfair. Plus, there are many reasons why a person could click on the button by mistake and share their data by accident. So even well-intentioned and savvy users may fall for it.

2. Developers And Designers

Let’s be honest. While the developers are not the root cause of the problem, they play a key role in exploiting the dark pattern. Either accidentally or with malicious intent.

And let’s be responsible and honest (this time in a literal way), because that’s the thing that developers and designers can do to build trust and make good use of the autofill and autocomplete features:

• Only auto-complete the data that you need.
• State clearly which data will be collected.
• Do not hide form fields that will be later submitted.
• Do not mislead or trick users into sending more data.

As an extreme measure, maybe try to avoid autocompleting certain fields. But, of course, this brings other issues as it will make the form less usable and accessible. So finding a balance may be tricky.

All this is without considering the possibility of an XSS vulnerability that could exploit the dark pattern. Of course, that would be a completely different story and an even more significant problem.

3. Browsers

Much of the work would need to be done from the browser side (especially on the Chromium side). But let me start by stating that not all is bad with how web browsers handle autofill/autocomplete. Many things are good. For example:

• They limit the data that can be shared
  Browsers have a list of fields for auto-complete that may not include all the values described in the HTML standard.
• They encapsulate and group data
  Browsers separate personal and financial information to protect critical values like credit cards. Safari had some issues with this, but it was minor.
• They warn about the data that will be shared
  Sometimes this may be incomplete (Chrome) or not clear (Safari), but they do alert the user.

Still, some things can be improved by many or all of the web browsers.

Show All Fields That Will Be Autocompleted

Browsers should always show a list of all the fields that will be autocompleted within the autofill popup (instead of just a partial list.) Also, the information should be clearly identified as data to be shared instead of being displayed as a regular contact card that could be misleading.

Firefox did an excellent job at this point, Safari did a nice job in general, and Chrome was subpar compared to the other two.

Do Not Trigger The onChange Event On Autofill

This would be a problematic request because this behavior is part of the Autofill definition in the HTML standard:

“The autocompletion mechanism must be implemented by the user agent acting as if the user had modified the control’s data [...].”

This means that browsers should treat the autocompleted data as if it had been entered by the user, thus triggering all the events, showing the values, etc. Even on a non-visually available field.

Preventing this behavior on non-visible elements could solve the problem. But validating if a form control is visible or not could be costly for the browser. Also, this solution is only partial because developers could still read the values even without the inputs triggering events.

Do Not Allow Devs To Read Autocompleted Fields Before Submission

This would also be problematic because many developers often rely on reading the field values before submission to validate the values (e.g., when the user moves away from the inputs.) But it would make sense: the user doesn’t want to share the information until they submit the form, so the browser shouldn’t either.

An alternative to this would be providing fake data when reading autocompleted values. Web browsers already do something like this with visited links, why not do the same with autocompleted form fields? Provide gibberish as the name, a valid address that matches local authorities instead of the user’s address, a fake phone number? This could solve the developer validation needs while protecting the user’s personal information.

Displaying a complete list of the fields/values that the browser will clearly share with the form would be a great step forward. The other two are ideal but more of stretch goals. Still, they are initiatives that would considerably improve privacy.

Would the autofill dark pattern still be possible to exploit? Unfortunately, yes. But it would be a lot more complicated. And at this point, it would be the user’s responsibility and the developer’s duty to avoid such a situation.

Conclusion

We can argue that autocomplete is not a huge security issue (not even on Chrome) because it requires user interaction to select the information. However, we could also argue that the potential loss of data justifies proper action. And Chrome has done more changes for (relatively) less important security/usability concerns (see alert(), prompt(), and confirm() than what could be done to protect key personal information.

Then we have the issue of the dark pattern. It can be avoided if everyone does their part:

• Users should be careful with what forms/data they autofill;
• Developers should avoid exploiting that data;
• Browsers should do a better job at protecting people’s data.

At the root, this dark pattern is a browser issue (and mainly a Chrome issue), and not a small one (privacy should be key online). But there is a choice. In the end, exploiting the dark pattern or not is up to the developers. So let’s pick wisely and do the right thing.

Further Reading

• Better Form Design: One Thing Per Page (Case Study), Adam Silver
• Common Concerns And Privacy In Web Forms, Vitaly Friedman
• Simplifying Form Styles With accent-color, Michelle Barker
• HTML5 Input Types: Where Are They Now?, Drew McLellan

(vf, yk, il, mrn)

We think of web scraping as a tool used to harvest web content for information analysis purposes, sometimes to the detriment of the site owner. For example, someone might web scrape all the product pages of a competitor’s retail site to harvest information about products being offered and current pricing to try to gain a competitive edge.

Web scraping can be used to harvest marketing data, such as identifying good keywords for advertising campaigns, identifying trending topics for blog posts, or identifying influencers in important blogs and news sites. Manufacturers might scrape retail sites to make sure that Manufacturer Advertised Pricing (MAP) is being adhered to, and security auditors might scrape sites to look for vulnerabilities or breaches in privacy policies. And of course, scrapers could scrape your site to look for security vulnerabilities or exposed contact or sales lead details. None of these activities would result in the scraped content being re-published or delivered to end-users.

There are situations, however, where the scraped web page content is delivered as a page itself directly to visitors. As we shall see below, this can be done for benign or not-so-benign reasons. In the worst-case scenarios, these can be true imposter domains, seeking to engage with real users without acknowledging the true source of your content. Even in benign use cases, however, you lose some control over the visitor’s experience. When your content is delivered by other means, from other servers or platforms, it can put the user experience and commercial relationship you have built up with your users at risk.

How can you identify, track, and manage this risk to your business? We explore how you can use web analytics or real user measurement data on your website to get insight into any imposter domains re-publishing your work. We also describe the most common types of content re-publishing we see in real-world data we collected in Akamai mPulse, both benign and problematic, so you know what to look for in your data.

How To Track Suspicious Activity

If you are just starting out wondering if someone might be re-publishing your web content, the easiest thing to do is a Google search. Copy a ten or twelve-word sentence from a page of interest on your site into the Google search bar, put it inside double-quotes, and click on Search. You should hopefully see your own site in the search results, but if you are finding that exact sentence matching on other sites, you may be the victim of re-publishing. This approach is obviously a bit ad-hoc. You could maybe script up some Google searches to run these kinds of checks periodically. But how many pages do you check? How can you reliably pick the content on the pages that the re-publishing is not going to modify? And what if the re-published page views never make it into Google’s search results?

A better approach is to use the data you are already collecting with your web analytics or Real User Measurement (RUM) services. These services vary considerably in their capabilities and the depth of data that is collected. They all generally are instrumented as JavaScript processes that are loaded onto your site’s web pages through a tag or snippet of loader code. When the service determines that a page view (and/or some other user activity of interest) is completed, it sends a “beacon” of data back to a collection system, where the data is then further processed, aggregated, and stored for future analysis.

To help identify the re-publishing of web pages by imposter domains, what you want is a service that:

• Collects data for every page view on the site (ideally);
• Collects the full URL of the base page HTML resource of the page view;
• Accepts beacons even if the hostname in that base page URL is not the one your site is published under;
• Allows you to query the collected data yourself and/or already has data queries designed to find “imposter domains”.

What Happens When A Web Page Is Re-Published?

When a web page is scraped with the intention of being delivered as a complete page view to an end-user, the scraper may modify the content. The modifications may be extensive. Modifying some content is easier than others, and while an imposter domain might change text or images, modifying JavaScript can be a more challenging proposition. Attempted modifications in JavaScript might break page functionality, inhibit proper rendering, or present other problems.

The good news for us is that web analytics trackers or real user measurement services are instrumented as JavaScript and many imposter domains are unlikely to try to modify the content to remove them because of the risks that it might break the page. If the scraper does not intentionally remove the loader snippet code or tag for your web analytics or RUM service, generally speaking, they will load successfully and generate a beacon for the page view — giving you evidence of the imposter domain activity.

This is the key to tracking imposter domains with web analytics or RUM data. Even if none of the page content is delivered from your platform or servers, so long as the JavaScript code you are using for analytics or performance tracking loads, you can still get data about the page view.

Turning The Data Into Information

Now that you have data, you can mine it for evidence of imposter domains. At the most basic, this is a database query that counts the number of page views by each hostname in the page URL, something like this pseudocode:

results = query("""
  select
    host,
    count(*) as count
  from
    $(tableName)
  where
    timestamp between '$(startTime)' and '$(endTime)'
    and url not like 'file:%'
  group by 1
  order by 2 desc
""");

Any hostname in the results that is not one your site uses is an imposter domain and worth investigating. For ongoing monitoring of the data, you will likely want to categorize the imposter domains you see in the data and have identified.

For example, some domains used by Natural Language Translation services that re-publish entire web pages might look like this:

# Translation domains
translationDomains = ["convertlanguage.com","dichtienghoa.com","dict.longdo.com",
  "motionpoint.com","motionpoint.net","opentrad.com","papago.naver.net","rewordify.com",
  "trans.hiragana.jp","translate.baiducontent.com","translate.goog",
  "translate.googleusercontent.com","translate.sogoucdn.com","translate.weblio.jp",
  "translatetheweb.com","translatoruser-int.com","transperfect.com","webtrans.yodao.com",
  "webtranslate.tilde.com","worldlingo.com"]

Depending on your needs, you might build up arrays of “acceptable” and “problem” domains, or categorize the imposter domains by their function or type. Below are the most common types of imposter domains you might see in real-world data.

Benign Re-publishing

Not all scraped web pages delivered from a third-party domain are going to be malicious. Based on looking at Akamai mPulse data across a broad spectrum of customers, most page views from imposter domains are actually services that a site visitor is intentionally choosing to use. A site visitor may be able to enjoy page content that they would find inaccessible. In some cases, the services are likely being used by the employees of the site owner itself.

The major categories described here are by no means exhaustive.

Natural Language Translation

The most common imposter domains are those used by natural language translation services. These services can scrape a web page, translate the encoded text on the page into another language and deliver that modified content to the end-user.

The page that the end-user sees will have a URL from the top-level domain of the translation service (such as translate.goog, translatoruser-int.com, or translate.weblio.jp among many others). rewordify.com changes the English text on a page into simpler sentences for beginning English speakers. While you have no control over the quality of the translations or the performance of the delivered page experience, it is safe to assume that most site owners would not consider this to be a business risk or concern.

Search Engine And Web Archive Cached Results

Another common category of imposter domains are domains used by search engines for delivering cached results or archived versions of page views. Typically, these would be pages that might no longer be available on the site but are available in third-party archives.

You might want to know about the frequency of these page views and deeper analysis could determine the specifics of what end-users were seeking in the online caches and archives. With the full URL for each request made to the online caches and archives, you should be able to identify keywords or topics that feature most often in those kinds of page views.

Developer Tools

These services will typically be used by your own employees as part of the natural business of developing and running your site. A typical developer tool might scrape an entire web page, analyze it for syntax errors in JavaScript, XML, HTML, or CSS, and display a marked-up version of the page for the developer to explore.

In addition to syntax errors, tools might also evaluate a site for compliance with accessibility or other legally required standards. Some example services seen in the real world include codebeautify.org, webaim.org, and jsonformatter.org.

Content Publishing Tools

Very similar to developer tools are tools that you might use to manage your content publishing needs. The most commonly seen example is the Google Ads Preview tool, which fetches a page, modifies it to include an ad tag and ad content from Google, and displays it to the site owner to see what the result would look like if published.

Another kind of content publishing tool is a service that fetches a web page, checks it against databases for any potential copyright violation or plagiarism, and displays the page with markup to identify any potentially offending content.

Transcoder Domains

Some services deliver a web page in altered form for either improved performance or improved display characteristics. The most common service of this type is Google Web Light. Available in a limited number of countries on Android OS devices with slow mobile network connections, Google Web Light transcodes the web page to deliver up to 80% fewer bytes while preserving a “majority of the relevant content” all in the name of delivering the content into the Android Mobile browser that much faster.

Other transcoder services modify the page content to change its presentation, e.g. printwhatyoulike.com removes advertising elements in preparation for printing to paper, and marker.to lets a user “mark up” a web page with a virtual yellow highlighter and share the page with others. While transcoder services can be well-intended, there is potential for both abuse (removing advertising) and potential questions of content integrity that you, as a site owner, need to be aware of.

Locally Saved Copies Of Web Pages

While not common, we do see beacons in the Akamai mPulse data with pages served from file:// URLs. These are page views loaded from a previously-viewed web page that was saved to device local storage. Because every device could have a different file system structure, resulting in an infinite number of “domains” in the URL data, it generally does not make sense to try to aggregate these for patterns. It is safe to assume that site owners would not consider this to be a business risk or concern.

Web Proxy Services

Another category of imposter domains that may be acceptable are those used by web proxy services. There are two large subcategories of presumed benign proxy services. One is institutional proxies, such as a university library system subscribing to an online news publication in order to grant access to its student body. When a student views the site, the page may be delivered from a host name in the university’s top-level domain.

It is safe to assume that most publishers would not consider this to be a business risk or concern if it is part of their business model. The other major type of benign proxies are sites that aim to offer anonymity so that visitors can consume a website content without being tracked or identified. The most common example of that latter subcategory is the anonymousbrowser.org service. The users of these services may or may not be well-intentioned.

Malicious Re-Publishing

While we have seen that there can be benign reasons for a web page to be scraped and then delivered from an alternative domain (and in fact, research shows that benign use cases are by far the most commonly seen in the Akamai mPulse real user measurement data), there are certainly instances where the intentions of the scrapers are malicious. Scraped content can be used to generate revenue in a variety of ways from simply passing off stolen content as one’s own to attempting to steal credentials or other secrets. Malicious use cases can harm both the site owner and/or the site visitor.

Ad Scraping

In the publishing industry, ad revenue is critical to the commercial success or failure of websites. Of course, selling ads requires content that visitors want to consume, and some bad actors may find it easier to steal that content than to create it themselves. Ad scrapers can harvest entire articles from a site and re-publish them on a different top-level domain with completely new advertising tags. If the scraper is not sophisticated enough to fully separate the content from the page structure, and for example includes core page JavaScript code including the loader snippet for your web analytics or RUM service, you can get beacons of data for these page views.

Phishing

Phishing is a fraudulent attempt to get users to reveal sensitive or private information such as access credentials, credit card numbers, social security numbers, or other data by impersonating a trusted site. To look as authentic as possible, phishing sites are often built by scraping the legitimate site that they aim to impersonate. Again, if the scraper is not sophisticated enough to fully separate the content from the page structure, and for example includes core page code including the loader snippet for your web analytics or RUM service, you can get beacons for these page views in mPulse.

Browser Or Search Hijacking

A web page may be scraped and re-published with additional JavaScript that contains browser or search hijacking attack code. Unlike phishing, which tempts users to surrender valuable data, this kind of attack attempts to make changes to the browser settings. Simply changing the browser’s default search engine to point to one that the attacker gains affiliate search result revenue from could be profitable for a bad actor. If the scraper is not sophisticated, injecting new attack code but not changing the pre-existing core page code including the loader snippet for your web analytics or RUM service, you can get beacons for these page views in mPulse.

Paywall Or Subscription Bypass Proxies

Some services claim to help end-users access pages on sites that require subscriptions to view without having a valid login. For some content publishers, subscription fees can be a very significant portion of site revenue. For others, logins may be required to remain in legal compliance for users to consume content that is restricted by age, citizenship, residency, or other criteria.

Proxy services that bypass (or attempt to bypass) these access restrictions pose financial and legal risks to your business. Subjectively, many of these services appear to be focused specifically on pornography sites, but all website owners should be on the lookout for these bad actors.

Misinformation

In addition to trying to profit off web scraping, some imposter domains may be used to deliver content that has been modified in a way to intentionally spread misinformation, harm reputations, or for political or other purposes.

Managing The Results

Now that you have a way to identify and track when imposter domains are re-publishing your website, what are the next steps? Tools are only as valuable as our ability to use them effectively, so it is important to develop a strategy for using an imposter domain tracking solution as part of your business processes. At a high level, I think this reduces to making decisions about a three-step management process:

1. Identify the threat,
2. Prioritize the threat,
3. Fix the threat.

1. Identifying Threats Through Regular Reporting

Once you have developed the database queries to extract potential imposter domain data from your web analytics or Real User Measurement data, you need to look at the data on a regular basis.

As a starting point, I would recommend a weekly report that can be quickly scanned for any new activity. A weekly report seems like the best cadence for catching problems before they become too severe. A daily report might feel tedious and become something easy to ignore after a while. Daily numbers can also be more challenging to interpret, as you can be looking at quite small numbers of page views that may or may not represent a concerning trend.

On the other hand, monthly reporting can result in problems going on for too long before being caught. A weekly report seems like the right balance for most sites and is probably the best starting cadence for regular reporting.

2. Categorization Of The Potential Threat

As we considered above, not all imposter domains re-publishing your site content are necessarily malicious in nature or a concern to your business. As you gain experience with the landscape of your own site’s data, you might enhance your regular reporting by color-coding or separating domains that you know about and consider non-malicious to help you focus on the unknown, new, or known problem domains that matter the most.

Depending on your needs, you might build up arrays of “acceptable” and “problem” domains, or categorize the imposter domains by their function or type (such as the “natural language translation” or “content publishing tools” categories described above). Every site will have different needs, but the objective is to separate the problematic domains from the domains that are not concerning.

3. Take Action Against The Bad Actors

For each of the problematic categories you identify, determine the parameters you want to use when deciding how to respond to the threat:

• What is the minimum page view count before we take action?
• What is the first point of escalation and who is responsible for it?
• Which stakeholders inside the business need to be aware of the malicious activity and when?
• Are the actions to be taken documented and reviewed by all stakeholders (executives, legal, etc.) on a regular basis?
• When actions are taken (such as filing a “DMCA Takedown” notice with the offender or their service provider or updating Web Application Firewall rules to try to limit access to the content thieves) are the results of these actions tracked and acknowledged?
• How will the effectiveness of these actions be summarized to executive business leaders over time?

Even if you are unsuccessful in squashing every malicious republication of your site content, you should still build a solid process in place to manage the risks like any other risk to the business. It will generate trust and authority with your business partners, investors, employees, and contributors.

Conclusion

In the right circumstances, your web analytics or real user measurement data can offer visibility into the world of imposter domains, used by web scrapers to re-publish your site content on their servers. Many of these imposter domains are actually benign services that either help end-users or help you in various productive ways.

In other cases, the imposter domains have malicious motives, either to steal content for profit or to manipulate it in a way that causes harm to your business or your site visitor. Web analytics or RUM data is your secret weapon to help identify any potentially malicious imposter domains that require immediate action, as well as to better understand the prevalence of the more benign domains. The data you collect leverages the web analytics or RUM service’s position as an observer in the visitor’s own browser to see that which your platform monitoring and reporting tools cannot.

Analyzing the data over time, you can learn more and more about imposter domains and their intentions in order to better inform your business about the risks they are posing to your reputations and your visitors’ experiences and develop and enforce mechanisms to protect your intellectual property.

Further Reading

• Protecting Your Site With Feature Policy
• Make Your Sites Fast, Accessible And Secure With Help From Google
• What You Need To Know About OAuth2 And Logging In With Facebook
• Content Security Policy, Your Future Best Friend
• Pushing Back Against Privacy Infringement On The Web

(vf, il, mrn)

At the ripe old age of 25, nytimes.com is now older than some of the people who work building it. Wikipedia turned 20 this year, and the first browser shipped 30 years ago. The Web has over 4 billion internet users and is nearing 2 billion websites. By this point, it’s reasonable to expect to have answers to most of the big, basic questions of how people and digital technology work with one another.

As you no doubt know, however, we don’t. We really don’t. If anything, questions of technology and society seem to be piling up faster than we can address them. I started working for The Times four years ago. Before that, I had done a range of tech work: developing sites at small web shops, product work with startups, consulting for household name tech companies, and heavy involvement in a range of W3C work. One of the things that drew me to The New York Times was the opportunity to work on some of those big unanswered questions in collaboration with excellent teams and with a strong ethical mandate.

Of these issues, perhaps the most surprising to find still unsolved is privacy. Outside of the digital realm, we make multiple privacy decisions a day and typically find them obvious enough that we barely notice they’re there. Is it okay to read that stranger’s DMs over their shoulder? Can I recount that intimate detail that a friend shared with me? Will my doctor repeat what I tell them of my symptoms to my boss?

Weighing up whether we — or someone we speak with — may take personal information from one context and share it in another is something we do almost instinctively. If we required expert opinion for every such decision, ethicist would be a high-income profession. So why can’t we seem to put this one to rest online as well?

The Digital Context

Several factors stand against us. The first is that privacy is contextual. We understand what may be shared to whom and how by using different frames for work, home, the subway, a doctor’s office, or the local neighborhood dive. But in our digital lives, everything is a slight variation on a glossy slab of plastic. We chat with friends on the same gadget we work on, speak in public through, and look up symptoms with. It is hard to develop cues for what counts as appropriate amidst such homogeneity.

A second factor is the usefulness of third parties. The modern development of complex digital products often requires relying on third parties one way or another. This is not necessarily bad. A company specializing in a clear service might have better data protection than a home-grown equivalent, and not all third parties are privacy-invasive. Some third parties work only for the first-party site and will not reuse user data elsewhere. For privacy purposes, they are indistinguishable from whatever the first party does. One example of this might be Fathom: it’s third-party analytics, but they have openly published the method they use to ensure user privacy.

Conversely, other third parties insist on being controllers of the data they take from the users of your site and reuse it independently for entirely different purposes. The latter are clear violations of privacy, but Web technology gives no way for browsers — and therefore users — to tell them apart. Cracking open your ad blocker extension and counting the “trackers” might be easy but it tells you little about how privacy-invasive a site really is. It is hard to automate protection at the browser level without the ability to tell third parties apart.

A final factor we shan’t be coy about is that there’s money to be made from this confusion. Many of the biggest names in tech (and a host of smaller ones too) operate with business models that do nothing other than convert privacy violations into money. They often muddy the waters with widely-read positions that extoll a confounding picture of privacy often confused with security, transparency, complex privacy settings and check-ups, or with consent. It can be hard to progress on informed improvements to privacy when so much of the conversation on the topic comes from a place of confusion.

This might feel like a lot to fix at once, but not to worry: we can all make the Web better one website at a time. It’s easy to become paralyzed thinking that all data collection is a privacy issue. That’s not the case. You can walk this path gradually. The way I approach privacy for a given site is to first try to find a familiar everyday context that I believe is close to it. Reasoning from known everyday contexts makes it possible to tap into established norms for which we tend to have good intuitions. Because the digital world works differently, there is rarely a perfect one-to-one match, but starting with a concrete situation can help structure your thinking.

Enter My Bookshop

Let us look at a bookshop and at how we can incrementally change its privacy properties by adding more details that render it increasingly like a modern commercial website. Hopefully, there is a point at which you will want to draw a line — and that point should tell you something about what is appropriate in a comparable context on the Web.

A bookshop makes for a decent real-world comparison point for an online publication. You go there looking for things to read. The people who work there can see you enter, and they might even recognize you from previous visits. If you’ve identified yourself, they may know your name, or at least have some moniker for you.

As you browse the shop, the staff can get a sense of what you’re looking at. They might not spy on it in great detail, but they’ll know which section you’re in and could easily notice which book you just pulled out. They can use that and ask you questions to offer recommendations.

It’s a commercial environment, and if you buy something there, depending on how you pay they might learn a little bit more about you, and share some of that data with a payment processing company of some sort.

At closer inspection, you notice that the shop has a few video surveillance cameras running. This makes you a little uncomfortable, but after a quick chat with the owner they reassure you that these are running on a local closed circuit such that the video never leaves the shop, is automatically kept for a maximum of 24 hours, and is only ever watched for the sole purpose of identifying theft. These properties — highly restricted access, short retention, clear and limited purpose — offer strong guarantees (assuming you trust the business) and limit how invasive the process is.

But, paying more attention, you notice a number of other smaller, more discrete cameras. They are dispersed throughout the store in a way that allows them to analyze which books you are looking at, which you pull out to read the blurb on, which you decide to buy.

The conversation with the bookshop owner is a bit more fraught this time. Those video feeds go to a number of companies in exchange for which they help pay a little of the shop’s costs or list the shop on neighborhood maps. You agreed to this by pushing the door open and stepping inside.

The owner is adamant that it’s safe, that the process is entirely “anonymous,” that these monitoring companies “only” use a hash of your facial biometrics to recognize you from shop to shop. What do you have to hide anyway? It’s an important part of making sure that the books stay cheap. Without that, only rich people would have books. Anything else would be bad for small businesses and poor people.

Flipping through the list of companies taking video feeds from this shop, you can’t find a name you recognize. Except for those two that seem to be taking video feeds from every single other shop as well.

Pushing the bookshop owner a little more, you come to realize they’re not very happy with those companies. They use the data they collect in the shop both to compete with the bookshop by selling books directly and to recommend other bookshops. They’re also using their position, informed by that data, to push for a model in which they run the infrastructure for all stores — why would a bookshop owner care about walls, shelves, lighting, is their value not just in the book selection? — and an increasing number of other shops have switched over to this worldwide strip mall model. The owner isn’t too sanguine about any of that. “But when you don’t comply they drop you off the map and send people to your competitors,” they say.

As promised, finding an exact non-digital equivalent to the state of privacy on the Web is imperfect and has some slightly contrived corners. Yet the above is very close to what it’s like to operate a commercial publisher or an online store on the Web today.

In reading the above, different people will draw the line at different places. Some are entirely bought into the death of privacy and look forward to the promised bright future of big tech bureaucracy. At the other end of the spectrum, others will prefer a staff-less bookshop where they can pick their next read with nary a human ever seeing what it may be.

If you’re like me, you’ll draw the line somewhere around the addition of the CCTV system. I’m not excited about it, but if its data processing is properly limited and if it helps my favorite source of books to stay alive, I can live with it. I am not inclined, however, to accept the pervasive surveillance of my behavior — the primary effect of which is for a handful of large corporations to either absorb small shops or drive them out of existence.

None of these positions is inherently right or wrong, but we help our users if we agree on a default and set expectations to match it. I believe that a good starting point for a default that works for most sites that perform a relatively small set of related services is the Vegas Rule: what happens on the site stays on the site. (Including the parties that work for the site, and only for the site.)

How Can We Build For Privacy?

What does this mean for those of us who build places on the Web that others visit? There is no one-size-fits-all answer to that. Last year, I wrote about how The Times sees it, but how you approach your own situation will depend on the specifics that you are dealing with.

I recommend working on privacy iteratively, with a plan to improve in gradual steps over time. In most cases, it’s unlikely that you’ll be able to deploy a catch-all Big Bang cleanup.

The first step towards making a site more trustworthy in its privacy is to understand what is going on and why. If you work on the tech or product side, you might have grown used to grumbling in dismay as the marketing team adds yet another tracker, which they refer to under harmless, cutesy euphemisms like “pixel,” that will slow down your site. But you might only have a limited understanding of why they do that in the first place.

These overlaps are great opportunities to build actual relationships with other teams. They are trying to achieve something, and that something is probably covering at least some of your pay-check. But they also sometimes struggle to tell honest vendors from hoaxes and they can benefit from a greater understanding of the technology that underlies what they buy — which you can likely help with. If you’re on their side instead of complaining about them, you might not be able to remove all the trackers, but you’ll be in a position to make progress.

Once you have that relationship (or if you’re the person making those decisions), the next step is to make sure that each third party is held accountable. The reality of online business today is such that you might have to keep some trackers, but those that stay should be provably effective. By working closely with our marketing team, we were able to reduce the amount of data The Times shared with third-party data controllers by over 90 percent. Not only did that improve privacy, but it also improved performance. The second step is to develop a habit of healthy skepticism.

Read the fine print so that your users don’t have to — it’s part and parcel of building a site that won’t betray your users’ trust.

“

That free widget to make it easy to share pages on social media? It’s probably a data broker. That comments system you can drop at the bottom of your blog? You might want to check that it’s not selling your users’ data. It’s not uncommon for a third party to strike deals with other third parties so that when you add one to your site it will inject the others as well, a practice known as piggybacking. It’s a good idea to run your site through a tool like Blacklight now and then to make sure that nothing surprising is going on (though if you run ads there will only be so much you can do, sadly).

Finally, you of course want to be thinking about your users, but you should also be thinking about yourself! The business implications of sending your audience data to third parties are often poorly understood and are too rarely taken into account. Understanding your users, your readers, your customers is a key business asset. When you share your audience data with third parties, they benefit from that too — and they can use that data to compete with you.

If you run a shoe store and let a social network track your users so you can target ads to them there later, know that that data, which reveals who is interested in shoes, will also be used to show them shoe ads for your competitors.

If you host your e-commerce pages on a virtual strip mall owned by a company that also runs its own online shopping business, how long will it be before they use what they learn from your customers to outcompete you?

Privacy isn’t just about ethics and earning the trust of your readers: it’s also a sound business strategy when, willingly or not, you participate in the data economy.

“

And How About We Fixed The Web, Too?

As a community, we can each work to improve our own little corners of the Web. But, collectively, we can do better. We can improve the Web and make it a better platform for privacy.

Perhaps the most important property of the Web is trust. You can browse around freely from site to site because you can trust that your browser will protect your security and that, even as those sites run code on your computer, they won’t endanger it. That is a strong promise, much stronger than what you can expect from native platforms, but when it comes to privacy it’s a promise that we have broken. As you browse freely from site to site, your privacy is not in trustworthy hands.

Still, things are looking up. Most browsers have delivered excellent work towards preventing tracking on the Web, and the biggest holdout, Chrome, has promised change in 2023. Projects like the Global Privacy Control are making headway. As third-party cookies slowly become extinct, industry stakeholders are working together to propose privacy-preserving standards that enable businesses to function on the Web without infringing on people’s privacy.

Detailing all the proposals on the table would require a whole other article, but Apple’s Private Click Measurement, Google’s FLEDGE, Microsoft’s PARAKEET, or, if you’ll allow me this shameless plug, The New York Times’s Garuda are all worth looking at, as is the work taking place in the Privacy CG. Some of the proposals discussed there, like Federated Learning of Cohorts (FLoC), have run into trouble, but that only underscores the value in building a solid understanding of privacy in the Web community in order to develop these novel solutions.

As much as we’d all love to engineer our way out of these complex problems, that’s rarely possible. Solutions will require cooperation between technologists and policymakers. As technologists, there are several ways in which we can help. We can act as citizens and take part in policy debates. We can describe our systems in understandable terms, and explain what is riding on them. (You’d be surprised at how many bad descriptions of cookies there can be.) We can use our understanding of technology to try to explore how what we build can create problems when deployed at large scales.

Most importantly, when technology fails society we can make it our responsibility to imagine other ways in which it could work — even if they seem out of immediate reach. Technology is often presented as inevitable as if its current design were the only rational option rather than the accumulation of arbitrary decisions that it is. We hold large, often untapped power in our ability simply to say that “it doesn’t have to be this way” and to show what other paths exist.

We build for users, not to milk them of their data. The Web has made it hard for them to stand for themselves — it’s on us to do it for them. Few of us who work on commercial sites will be able to produce perfect privacy outcomes immediately, but this should not stop us from doing better. The tide has turned and a privacy-friendly Web now seems possible. We might yet get to check one of those big, basic questions off the list.

Other Resources

• A great book to read to understand how privacy works is Helen Nissenbaum’s Privacy in Context. A shorter overview is available in Matt Salganik’s Bit by Bit which you can read online.
• Privacy is Power by Carissa Véliz offers an effective and well-documented overview of why privacy matters.
• James C. Scott’s classic Seeing Like A State presents a dire overview of how society-level engineering can fail, often in catastrophic ways, which includes an extensive description of the problems brought about by “legibility”, which is what collective infringement of privacy leads to.
• Maria Farrell’s This Is Your Phone On Feminism is an outstanding description of how we love our devices but don’t trust them, and of how love without trust is the definition of an abusive relationship.
• The W3C’s Technical Architecture Group (TAG) is working on a set of privacy definitions to help support discussions in the Web community. The output from that isn’t baked yet, but it is being built based on a blend of the Principles of User Privacy (PUP) and the Target Privacy Threat Model.
• In this article, I could only cover some of the basics, but one important concern in privacy is how data collection affects us at the collective instead of the individual level that we too often fixate on. Salomé Viljoen’s Democratic Data covers that issue very well.
• Woodrow Hartzog and Neil Richards have written so many excellent articles on privacy that it’s hard to pick a favorite, but I would suggest Taking Trust Seriously In Privacy Law as a great place to start. Don’t be daunted by the “law” part, it’s really about trust.
• Looking at a broader issue, Re-Engineering Humanity, by Brett Frischmann and Evan Selinger, is a great description of how we can misuse technology to shape people against their interests.
• More on the advertising side, Subprime Attention Crisis by Tim Hwang explains why we should worry about online advertising and the businesses it supports.

Further Reading

• Fine-Grained Access Handling And Data Management With Row-Level Security
• The Potentially Dangerous Non-Accessibility Of Cookie Notices
• Passkeys: A No-Frills Explainer On The Future Of Password-Less Authentication
• Everything I Know About UX Research I First Learned From Lt. Columbo

(vf, il, mrn)

Last week, I gave you an update on everything that’s happened with GDPR since 2018. (TL;DR: A lot has changed.) In this article, we’ll look at cookie consent: specifically, the paradox where marketers are heavily reliant on Google Analytics cookie data but need to comply with regulations.

We’ll take a look at two developments that have impacted cookies, plus a third on the horizon. Then I’ll walk you through the risk-based approach that we’ve taken — for the moment, at least. And come back next time for a deep dive into first-party ad tracking as we start to see moves away from third-party cookies.

• Part 1: GDPR, Key Updates And What They Mean
• Part 2: GDPR, Cookie Consent And 3rd Parties
  

Big Development #1: Cookie Consent

In May 2020, the EU updated its GDPR guidance to clarify several points, including two key points for cookie consent:

• Cookie walls do not offer users a genuine choice, because if you reject cookies you’re blocked from accessing content. It confirms that cookie walls should not be used.
• Scrolling or swiping through web content does not equate to implied consent. The EU reiterates that consent must be explicit.

What does this mean for our industry?

Well, the EU is tightening up on cookie consent — perhaps the most noticeable (and annoying!) aspect of GDPR. Critics say that cookie notices are a cumbersome block for users, and don’t do anything to protect user privacy. The EU is trying to change this, by promoting simple, meaningful, equitable options for cookie consent.

But that restricts what we can do with cookies, and it hints ahead to when the Privacy and Electronic Communications Regulation (PECR) may come into force. More on that shortly.

Big Development #2: Google And Apple Crack Down On Third-Party Tracking; Get Hit By Anti-Trust Complaints

As the big digital players figure out how to comply with GDPR — and how to turn privacy legislation to their advantage — some have already come under fire.

Google is being investigated by the UK’s competition watchdog, the Competition and Markets Authority (CMA), for its ‘Privacy Sandbox’ initiative, following complaints from adtech companies and publishers.

The Internet giant, which is also facing an antitrust investigation in Italy for display advertising, and in the US for its search advertising services, is looking to remove third-party cookies from Chrome. (Firefox and Safari already block these cookies by default.)

The complainants say that this change will further concentrate advertising revenue in Google’s hands. Google’s response? The advertising industry needs to make ‘major changes’ as it shifts to a ‘web without third-party cookies’.

Google’s not alone. In October 2020, four French digital advertising lobbies filed an antitrust suit against Apple’s forthcoming iOS privacy change, a feature it’s called App Tracking Transparency (ATT).

ATT, coming in an early-spring 2021 release of iOS 14, shifts app users from an opt-out to an opt-in ad-tracking model. With ATT, every app must get your permission to share your Identifier for Advertisers (IDFA), which enables third-party ad tracking across multiple sites and channels.

The complainants say that by restricting apps’ ad revenue, developers may have to boost app subscriptions and in-app purchases or switch to Apple’s targeted ad platform — all of which will funnel ad spend away from them and towards Cupertino.

Critics including Facebook have slammed the change, saying it’ll hit small businesses who rely on microtargeted ads. Apple has defended the move and praised the EU’s defence of citizens’ data privacy.

To sum up:

• Implied consent doesn’t equal consent under GDPR, according to the EU.
• We should also avoid cookie walls
• Google and Apple are moving against third-party cookies — which some say exploits their dominant market position.

So what does that mean for us, as designers and developers? First, let’s take a look at why this is important.

Here’s What Designers Should Know About Cookies

• GDPR is critical for you because you’ll design the points at which cookies are placed, what data is collected, and how it’s processed.
• A functionality audit means you can map your cookie activity in the data and compliance layers on your service blueprint.
• It can help to do a cookie audit and gap analysis, i.e. is the existing cookie pattern compliant? What content does it need around it?
• Follow Privacy by Design best practices. Don’t try to reinvent the wheel — if you’ve created a compliant cookie banner, use your proven design pattern.
• Work with your compliance and development teams to ensure designs meet GDPR and can be implemented. Only ask for the data you need.
• If you need to compromise, take a risk-based approach. There’s a walk-through of one that we did further down.
• Be aware that your content team may need to update your privacy policy as GDPR and your use of cookies evolve.

Here’s What Developers Should Know About Cookies

• Make sure you’re involved upfront about cookie consent and tracking, so what’s decided can be implemented.
• If you’re doing a product or website redesign, a cookie audit using Chrome Dev Tools can show you what tracking cookies are being used. Tools like Ghostery or Cookiebot give you more detail.
• You should implement the standard cookie opt in/out as per GDPR guidance. (Note that while GDPR is standard, the enforcement of it varies across EU countries. There’s more on this further down.) You may stand to lose Google Analytics data. You might also come under pressure to implement things that could be considered as dark patterns. There’s more on this later, with a walk-through of what we did and a look at the risk.

So that’s where we are today. Oh, and there’s one more thing to be aware of: a piece of further legislation that might be coming our way. I like to call it Schrodinger’s Law.

Schrodinger’s Law: The ePrivacy Regulation

You may have heard of GDPR’s twin sister, the ePrivacy Regulation, who’s lurking on the legislative horizon. If you haven’t, here’s an introduction.

As I said above, cookie consent — the notice that pops up when you visit a website — is regulated by the GDPR. However, cookies themselves fall under a different piece of legislation, the ePrivacy Directive of 2002, commonly known as the Cookie Law. Like GDPR, it aims to protect customer privacy.

The ePrivacy Directive is due to be replaced by more stringent legislation, the ePrivacy Regulation. (If you’re interested in the difference between EU directives and regulations, EU directives set out the goals for legislation but delegate the implementation of those goals to member states’ legislatures. EU regulations mandate both the goals and the implementation at an EU-wide level.)

The draft ePrivacy Regulation goes beyond cookies and ad tracking. It applies to all electronic communications, including messaging apps, spam mail, IoT data transfer and more.

“

The draft ePrivacy Regulation was first presented by the EU in 2017. However, it has to be agreed by both the European Parliament and the Council of the European Union. (The Council consists of government representatives of each EU member state.)

This is where it gets messy. Since 2017, the European Parliament and the Council haven’t been able to agree on the scope and detail of the ePrivacy Regulation.

That’s because some countries — widely thought to include the Nordic states of Finland and Denmark — want to strengthen the current ePrivacy Directive. They want users, for example, to be able to set acceptance and rejection of tracking cookies in their browsers, not on every site they visit.

But other countries, notably Austria and believed also to include those with sizeable digital marketing and advertising sectors, say this is bad for business. It’s thought the 27 EU member states are split down the middle on this issue — and they’re all being heavily lobbied by the tech industry.

So the draft regulation has been ricocheting back and forth between the European Commission and its Working Party on Telecommunications and Information Society as they try to agree its scope. In November 2020, the Working Party rejected the redrafted legislation once again.

What happens next? There are two possibilities. Either a compromise will be reached, in which case the legislation will be agreed. Because it takes time for legislation to be implemented, the soonest the ePrivacy Regulation could become law is 2025.

Alternatively, the legislation cannot be agreed and is withdrawn by the European Commission. But the EU has staked so much on it. It will be extremely reluctant to take that step.

That’s why I call it Schrodinger’s Law. It’s hard for us to know how to plan for any cookie-related developments because we simply don’t know what’s happening.

So What Should I Do About Cookies Right Now?

Different EU countries are currently implementing the ePrivacy Directive differently. Over in the UK, the ICO (the UK’s data protection authority) is taking a tough stance. It’s requiring strict consent for analytics cookies, for example, and has spoken out against cookie walls.

Until — and if — we get consistency from a new ePrivacy Regulation, if you’re based in an EU country, start by following the advice from your national Data Protection Authority. Then watch this space for developments around the ePrivacy Regulation.

If you’re based outside the EU, make sure you’re giving EU citizens the options required under the GDPR and the ePrivacy Directive.

“

However, when it comes down to the detail, there are times when I recommend taking a risk-based approach. That’s what we’ve done at Cyber-Duck — and here’s why.

Here’s our original cookie notice. You see these everywhere. They’re pretty meaningless — users just hit accept and continue on their way.

But we wanted to be compliant, so we replaced it with this notice. You’ll see that tracking cookies are turned off by default — in line with ICO guidance. We knew there was a risk we would lose analytics data as GTM would no longer fire on first load.

Let’s see what happened.

Problem solved? Actually, no. It just created another problem. The impact was far more significant than we expected:

Look at the collapse in the blue line when we implemented the new cookie notice. We released the new cookie consent on 17 December and went straight from plenty of tracked traffic to almost zero. (The orange line shows the previous year’s traffic, for comparison.)

In both the before-and-after scenarios, the default option was by far the most popular. Most users just naturally click on “accept” or “confirm”. That’s tricky, because we now know so little about the people visiting our site that we can’t give them the best information tailored to their needs.

We needed a solution. Analytics and marketing data ultimately drive business decisions. I’m sure we all know how important data is. In this case, it was like putting money in a bank account and not knowing how much we’d spent or saved!

Some of the solutions that were posed include design alternatives (would removing the toggle, or having two buttons with a visual nudge towards the “accept” help?) Or would we enable analytics cookies by default?

For now, we’ve implemented a compromise position. Marketing and analytics cookies are on by default, with one clear switch to toggle them off:

And here’s what that’s done to our stats:

The new cookie banner was relaunched on 15 January. You can see our website traffic starts to pick back up again. However, we’re not getting the full data we were getting before as Google Tag Manager doesn’t fire unless a user chooses cookies.

The good news is, we are getting some data back again! But the story doesn’t end here. After we had turned cookie tracking back on by default, the attribution model got messed up. It wasn’t attributing to the correct channel in Google Analytics.

Here’s what we mean:

Scenario 1: (Correct Attribution)

1. User lands on our website via a paid ad (PPC) or from the search result (organic)
2. User accepts cookies straight away.
3. The channel source is attributed correctly, e.g. to PPC.

Scenario 2: (Incorrect Attribution)

1. User lands on our website via a paid ad (PPC) or from the search result (organic)
2. User visits a few other pages on our website without responding to the cookie banner prompt (banner appears on every page until it gets a response)
3. User finally accepts cookie banner after browsing a few pages.
4. Attribution comes through as direct — although they originally came from a search engine.

How does that work? When a user browses other pages on the site, nothing is tracked until they respond to the cookie prompt. Tracking only kicks in at that point. So to Google, it looks as though the user has just landed on that page — and they are attributed to Direct traffic.

Back to the drawing board.

Note: I’m sure by now you’re starting to see a pattern here. This entire experience is new for us and there’s not a lot of documentation around, so it’s been a real learning curve.

Now, how could we solve this attribution issue and stop users from navigating around the site until they’ve selected their cookie preference?

A cookie wall is one option we considered, but that would potentially push us further away from being compliant, according to the ICO. (Though you might like to try browsing their site incognito and see if they stick to their own guidance…)

But that’s what we’ve chosen to go with. The journey ends here for now, as we’re still gathering data. In the future, we want to explore other tools and the potential impact of moving away from Google Analytics.

So what’s everyone else doing?

Well, McDonald’s UK offers straightforward on/off buttons:

Coca Cola’s British site nudges you to accept by making the ‘reject’ option harder to find:

Whereas Sanrio just has an option to agree to ad tracking:

Hello Kitty, hello cookies.

Die Zeit offers free access if you accept tracking cookies — but for an untracked, ad-free experience you’ll have to pay:

And here’s one of my favourite dark patterns. This restaurant site only has the ‘Necessary’ cookies selected. But it nudges you to the ‘Allow all cookies’ big red button — and when you click that, the analytical and ad cookie boxes are automatically checked and set. Give it a go here!

Even the EU isn’t consistent on its own sites.

The European Parliament’s cookie consent offers two clear options:

The CJEU’s site isn’t so clear:

While Europol’s site comes with two pre-checked boxes:

And if you look at the sites for the German presidency of the Council of the European Union (July–December 2020), at first it seems as if there’s no cookies at all:

When you land on the site, there are no cookie banners or prompts. A closer look, with cookie extension tools, shows that no cookies are being placed either.

So are they capturing any analytics data? The answer is yes.

We found this little snippet in their code, which shows they are using ‘Piwik’. Piwik is now known as Matomo, one of a clutch of new tools that help with cookie compliance along with Fathom (server-side tracking) and HelloConsent (cookie management).

So alternatives and solutions are emerging. We’ll take a closer look at that next time — with new alternatives to third-party cookies that will help you take control of your data and get the insight you need to deliver optimum experiences to your customers. Stay tuned!

Other Resources

• Data protection (the EU’s site)
• UK ICO’s Guidance On Cookies
• GDPR Enforcement Tracker, logs fines applied under GDPR
• GDPR Checklist, by Cyber-Duck — a great place to start
• Overview of Data Protection Law in the United States, by ICLG
• GDPR & CCPA Comparison Guide, by DataGuidance and the Future of Privacy Forum
• CCPA vs CPRA, from IAPP
• Security By Design (Amazon)
• How To Protect Your Users With The Privacy By Design Framework, Heather Burns, Smashing Magazine

Further Reading

• How GDPR Will Change The Way You Develop
• Privacy By Design: How To Sell Privacy And Make Change
• Fine-Grained Access Handling And Data Management With Row-Level Security
• The Potentially Dangerous Non-Accessibility Of Cookie Notices

(vf, il, mrn)

The EU’s directives have impacted virtually every digital professional as products and services are designed with GDPR in mind, regardless of whether you’re a web design company in Wisconsin or a marketer in Malta. The far-reaching implications of GDPR don’t just impact how data should be processed, how products should be built and how data is transferred securely within and between organisations. It defines international data transfer agreements like that between Europe and America.

Kevin Kelly, one of the world’s brightest digital futurists, claims that ‘Technology is as great a force as nature’. What he means by that is that user data and information technology is causing one of the most profound periods in human history since the invention of language. Just look at what is happening as governments and the tech multinationals grapple to control the Internet.

Last week alone, as the Australian government moved to force platform owners to pay publishers for the content that’s shared on their platform, Facebook decided to block news to Australian users with a huge uproar from the Australian government.

And that’s in addition to previous controversies (the organisation of the U.S. Capitol riot, the Cambridge Analytica scandal) at the intersection where government and technology meet.

In this article, we’ll look at how GDPR has evolved since 2018. We’ll run through some updates from the EU, some key developments, and where GDPR is likely to evolve. We’ll explore what that means for us, as designers and developers. And we’ll look at what that means for companies both inside and outside the EU.

In the next article, we’ll focus on cookie consent and the paradox where marketers are heavily reliant on Google Analytics cookie data but need to comply with regulations. And then we’ll take a deep dive into first-party ad tracking as we start to see moves away from third-party cookies.

• Part 1: GDPR, Key Updates And What They Mean
• Part 2: Cookie Consent For Designers And Developers
  

A Quick Recap Of GDPR

Let’s start by reminding ourselves what GDPR is. The GDPR became law within the EU on 25 May 2018. It’s based on 7 key principles:

1. Lawfulness, fairness and transparency
   You must process data so that people understand what, how, and why you’re processing their data.
2. Purpose limitation
   You should only collect data for clear, specified, and legitimate purposes. You can’t then process it in ways that are incompatible with your original purposes.
3. Data minimization
   You should only collect the data you need.
4. Accuracy
   Your data must be accurate and kept up to date. Inaccurate data should be erased or corrected.
5. Storage limitation
   If data can be linked to individuals, you can only keep it for as long as you need to carry out the purposes you specified. (Caveats for scientific, statistical, or historical research use.)
6. Integrity and confidentiality (i.e. security)
   You must ensure the personal data you hold is processed securely. You must protect it from unauthorized or unlawful processing and against accidental loss, destruction, or damage.
7. Accountability
   You are now responsible for the data you hold and should be able to demonstrate your compliance with the GDPR.

Some Definitions

• CJEU
  Court of Justice of the European Union. This court’s decisions clarify EU laws like GDPR.
• DPAs
  National Data Protection Authorities. Each EU country has one. GDPR is enforced, and fines are issued, at the national level by these bodies. The UK equivalent is the Information Commissioner’s Office (ICO). In the United States, GDPR-style data privacy is largely legislated by each state.
• European Commission
  The executive branch of the European Union (essentially the EU’s civil service). The European Commission drafts legislation including the GDPR.
• GDPR
  The 2018 General Data Protection Regulation.

Key Updates From The EU

GDPR hasn’t stood still since May 2018. Here’s a quick run-through of what’s happened since it came into effect.

How Have The EU And Its Member States Implemented GDPR?

The European Commission reports that GDPR is almost fully implemented across the EU, though some countries — it namechecks Slovenia — have dragged their feet. However, the depth of implementation varies. The EU also says its member countries are, in its opinion, using their new powers fairly.

However, it has also expressed concern that some divergence and fragmentation are creeping in. GDPR can only work effectively across the EU’s single market if member states are aligned. If the laws diverge, it muddies the water.

How Does The EU Want GDPR To Develop?

We know the EU wants it to be easier for individuals to exercise their rights under GDPR. That means cross-border collaboration and class-action lawsuits. It wants to see data portability for consumers beyond banking and telecoms.

It also wants to make it easier for small and medium-sized enterprises (SMEs) to comply with GDPR. That’s likely to come in the form of extra support and tools such as more standard contractual clauses — essentially templated legalese that SMEs can copy/paste into contracts — as the EU isn’t keen to bend the rules for them.

Big Development #1: The Unexpectedly Broad Definition Of ‘Joint Controller’

Right, here’s the first big change since GDPR became law. In two test cases involving Facebook, the Court of Justice of the European Union has defined a far broader interpretation of ‘joint controller’ than expected.

A joint controller situation arises when two or more controllers both have responsibility for meeting the terms of the GDPR. (Here’s a good explainer from the ICO on joint controllers.) Essentially:

• When you process customer data, you decide with your fellow joint controller(s) who will manage each step so you’re compliant with the GDPR.
• However, you all have full responsibility to ensure the entire process is compliant. Each of you is fully accountable to the data protection authority in the country handling any complaints.
• An individual can raise a complaint against each and all joint controllers.
• You are all responsible for any damage caused — unless you can prove you have no connection to the event that’s caused the damage.
• An individual can seek compensation from any joint controller. You may be able to reclaim some of that compensation from your fellow controllers.

In the first Facebook case, the CJEU confirmed that a company that ran a Facebook fan page counted as a joint controller alongside Facebook. In the second, the CJEU also confirmed that a company that embedded a Facebook Like button onto its website held joint controller status with the social network.

These cases sent shockwaves through the privacy community, as essentially it makes social publishers, website operators, and fan page moderators responsible for user data alongside platforms like Facebook.

However, the CJEU also clarified that shared responsibility does not mean equal responsibility. In both cases, responsibility sat primarily with Facebook — only Facebook had access to the data and only Facebook could delete it. So the impact of this decision may be less severe than it sounds at first — but it’s still critically important.

And that might be why some sites — such as the website for Germany’s 2020 presidency of the EU — block embedded social content by default, until you’ve specifically opted in:

Big Development #2: Bye Bye Privacy Shield, Hello CPRA

The second big change was more predictable: Privacy Shield, the mechanism that made it easier for American businesses to process European customer data, has been struck down by the courts.

Here’s why.

The EU wants to protect its citizens’ personal data. However, it also wants to encourage international trade, plus cross-border collaboration in areas like security.

The EU sees itself — quite rightly — as a pioneer in data protection. So it’s using its political muscle to encourage countries who want to trade with the bloc to match its data privacy standards.

“

Enter the United States. European and American philosophies around data privacy are diametrically opposed. (In essence, the European view is that personal data is private unless you give explicit permission. The American view is that your data is public unless you expressly request that it’s kept private.) But as the world’s two biggest consumer markets, they need to trade. So the EU and the US developed Privacy Shield.

Privacy Shield was designed to enable US companies to process EU citizens’ data, as long as those companies signed up to its higher privacy standards.

But under US law, the US government could still monitor that data. This was challenged in a case brought by Austrian privacy advocate Max Schrems. The CJEU sided with him: Privacy Shield was struck down and the 5,300 American SMEs who used Privacy Shield were given no choice but to adopt the EU’s prescribed Standard Contractual Clauses.

Obviously, it’s in everyone’s interests for Privacy Shield to be replaced — and it will be. But experts say that its replacement is likely to be struck down again in due course because European and American approaches to privacy are essentially incompatible.

Meanwhile, in California, 2018’s GDPR-inspired California Consumer Privacy Act (CCPA) was strengthened in November 2020 when the California Privacy Rights Act (CPRA) was passed.

The California Consumer Privacy Act (CCPA)

The CCPA, which came into effect in January 2020, gives California citizens the right to opt out of their data being sold. They can also ask for any data that’s been collected to be disclosed and they can ask for that data to be deleted. Unlike GDPR, the CCPA only applies to commercial companies:

• Who process the data of more than 50,000 California residents a year, OR
• Who generate gross revenue of more than $25m a year, OR
• Who make more than half of their annual revenue from selling California residents’ personal data

The California Privacy Rights Act (CPRA)

The CPRA, which comes into force in January 2023, goes beyond the CCPA. Its key points include:

• It raises the bar to companies that process the data of 100,000 California residents a year
• It gives more protection to Californians’ sensitive data, such as their race, religion, sexual orientation, and health data and government ID
• It triples the fines for breaches of minors’ data
• It gives Californians the right to request their data is corrected
• It obliges companies to help with CPRA investigations
• And it establishes a California Privacy Protection Agency to enforce the CPRA

Further pushes towards privacy laws are happening in other states, and together these may reinforce the need for federal privacy measures under the new Biden administration.

Big Development #3: Cookie Consent

In May 2020, the EU updated its GDPR guidance to clarify several points, including two key points for cookie consent:

• Cookie walls do not offer users a genuine choice, because if you reject cookies you’re blocked from accessing content. It confirms that cookie walls should not be used.
• Scrolling or swiping through web content does not equate to implied consent. The EU reiterates that consent must be explicit.

I’ll be going deeper into this in the second article next week.

Big Development #4: Google And Apple Start To Shift From Third-Party Tracking

As the big digital players figure out how to meet GDPR — and how to turn privacy legislation to their advantage — some have already come under fire.

Both Google and Apple are facing antitrust lawsuits, following complaints from adtech companies and publishers.

In both cases, the complainants says the big tech companies are exploiting their dominant market position.

Again, more on this next time.

Big Development #5: Big GDPR Fines Coming This Way

Of course, many organizations jumped to comply with GDPR because they feared the fines that regulators could apply. Those fines have started rolling in:

The French data regulator has slapped Google with a €50m fine for “lack of transparency, inadequate information and lack of valid consent regarding ads personalization”, saying users were “not sufficiently informed” about how and why Google collected their data.

Its UK equivalent, the ICO, has fined US hotel conglomerate Marriott International Inc. £18.4m for failing to keep 339 million guest records secure. The 2014 cyber-attack on Starwood Hotels and Resorts Worldwide, Inc., which Marriott acquired in 2016, wasn’t discovered until 2018.

The UK’s ICO has also fined British Airways a record £20m for a 2018 data breach of 400,000 customers’ personal and credit card data.

Then there’s my personal favorite, a shocking breach of employee trust by H&M that led to a €35m penalty.

So that’s where we stand today.

What Does This Mean For You?

As designers and developers, GDPR has — and will continue to have — a big impact in the products we design and build, and the way that we design for data.

Here’s What We, As Designers, Should Know

• GDPR is critical for you because you’ll design the points at which users share their data, what data is collected, and how it’s processed.
• Follow Privacy by Design best practices. Don’t try to reinvent the wheel — if you’ve created a compliant cookie banner, use your proven design pattern.
• Work with your compliance and development teams to ensure designs meet GDPR and can be implemented. Only ask for the data you need.
• Finally, ask your users what data they’re comfortable sharing and how they’d like you to use it. If they find it creepy, revisit your approach.

Here’s What We, As Developers, Should Know

• GDPR is critical for you because you enable data processing, sharing and integrations.
• As a general rule with GDPR, take a need-to-access approach. Start by implementing everything with no access, then only give your team access to data as and when it’s necessary (e.g. giving developers access to the Google Analytics console). Audit and document as you go.
• Follow privacy by design and security by design principles. Robust, secure templates for implementing infrastructure are key.
• Make sure you’re involved upfront about technical aspects e.g. cookie consent/tracking conversations, so what’s decided can be implemented.
• Process mapping shows where data is being shared with different parts of the business.
• Automation offers secure data handling that cuts human error. It also helps prevent the wrong people accessing data.
• GDPR checklists and of course run books will help you manage your process. Again, audit and document as you go.

Now let’s see how GDPR is going to evolve in the near future. We’ll focus on three areas.

Three Areas Where GDPR Is Swiftly Evolving

1. How The EU Is Implementing GDPR

First up, let’s see how GDPR will be further embedded in the legislative landscape.

The EU wants to keep its member states aligned, because that will make cross-border suits and international collaboration easier. So it has reinforced that countries should neither divert from, nor overstep the GDPR. Some member states, as I said, are paying lip service to the regulation. Others want to exceed GDPR’s standards.

In return for their alignment, the EU will enforce compliance, work to enable class action and cheaper cross-border suits, and also promote privacy and consistent standards outside the EU. In addition to extra support and tools for SMEs, we may also see certification for security and data protection by design.

Finally, this could raise some eyebrows in Silicon Valley: the EU has hinted that it might consider bans on data processing to encourage compliance. €50m fines aren’t the end of the world for Google and friends. But time out on the naughty step — and the resulting bad PR — is a very different thing.

2. How GDPR Works With Innovation

GDPR was designed to be technology-neutral and to support, not hinder, innovation. That’s certainly been tested over the past 12 months, and the EU points to the rapid rollout of COVID-19 apps as proof that its legislation works.

We can expect to see codes of conduct for sensitive categories of data (health and scientific research). These will be welcomed.

However, they’re watching innovators closely. The EU has expressed concern about data privacy in video, IoT devices and blockchain. They are particularly concerned about facial (and presumably voice) recognition and developments in AI.

Most notably, the Commission is deeply concerned about what it calls “multinational technology companies”, “large digital platforms” and “online advertising and micro-targeting”. Yes, once again it’s looking at you, Facebook, Amazon, Google and friends.

3. How The EU Is Promoting GDPR Standards Beyond The EU

Our digital economy is global, so GDPR’s impact ripples beyond the EU’s borders — and not just in terms of compliance. The EU is setting the bar for data protection legislation worldwide. Beyond California’s CCPA, see Brazil’s LGPD, plus developments in Canada, Australia, India and a clutch of American states.

Of course, it’s in the EU’s interests if other countries and trading blocs match their standards. So it’s promoting GDPR via several avenues:

• Through “mutual adequacy decisions” with Japan and shortly South Korea
• Embedded into bilateral trade agreements e.g. with New Zealand, Australia, UK
• Through fora like the OECD, ASEAN, the G7 and the G20
• Through its Data Protection Academy for EU and international regulators

It is particularly keen to empower innovation through trusted data flows and to enable international cooperation between law enforcement authorities and private operators.

The EU is leading the world in data protection. Where it goes, others will follow. So even if you’re not designing/developing for an EU audience, you need to be aware of what’s happening.

“

What Does All Of This Mean For Companies In The EU?

Companies who operate in the EU need to comply with GDPR or risk being fined. Those fines can be pretty hefty, as we’ve seen. So you need to be able to demonstrate that you’re adhering to GDPR’s 7 principles and to specific guidance from your national Data Protection Authority.

However, that’s not as straightforward as it sounds, and you may choose to evaluate your risk in some cases. I’ll take you through an example of that next time.

What Does This Mean For Companies Based Outside The EU?

The implications for companies based outside the EU are exactly the same as those for EU countries, if they process personal data from the EU. That’s because GDPR applies to the personal data of people based in the EU. If you want to process it, e.g. to sell to customers in the EU, you have to abide by the rules. Otherwise, you risk being fined, like Facebook and Google.

Here’s how that’s enforced: If you have a presence in the EU, as many multinationals do, and you don’t pay a GDPR fine, your EU assets may be seized. If you don’t have a presence, you’re obliged under GDPR to appoint a representative in the EU. Any fines will be levied through that representative. Alternatively, you may face a complex and expensive international lawsuit.

And here’s where it gets complex for everyone:

If your customer base includes people in the EU and citizens of other places with privacy laws, such as the State of California, you have to comply both with the California Consumer Privacy Act (CCPA) and with GDPR. These batches of legislation generally align — but they don’t match.

Take cookies, for example. Under GDPR, you must get active consent from a user before you place a cookie on their device, bar those strictly needed for your site to function.

However, under the CCPA, you must disclose what data you’re collecting, and enable your customer to deny you permission to sell their data. But they don’t have to actively agree you can collect it.

That’s why the EU is pushing for international standards to simplify global compliance.

N.B. If you’re in the United States and eagerly awaiting the replacement to Privacy Shield, you might like to take a leaf from Microsoft’s book instead — they and others have stated they’ll comply with GDPR rather than depend on any bilateral mechanisms to enable data processing.

What Lessons Can Web Designers And Developers Learn From GDPR?

Privacy regulation is here to stay and it affects all our priorities and workflows. Here are six lessons to remember as you work with customer data:

1. We had to sprint to comply with GDPR. Now it’s a marathon.
   We know that GDPR will continue to evolve alongside the technology it aims to regulate. That means the demands on us won’t remain the same. Not only that, but GDPR has inspired similar — but not identical — legislation around the world. These legal requirements are set to keep evolving.
2. Compliance builds competitive advantage.
   While the first major GDPR fines have been eye-watering, it’s actually the negative publicity that many say is most damaging. Who benefits from a large data leak? The company’s competitors. On the other hand, if you embed GDPR compliance as you strengthen your design and development processes, you’ll be better able to adapt as the regulations evolve.
3. GDPR compliance and better COVID-19 outcomes are linked by user-centred design.
   We know that companies who’d begun their digital transformation were better able to adapt to the COVID-19 crisis. User-centred design supports GDPR, too. It has the process and customer focus you need to build products that align with the idea that customer data is precious and must be protected. That will make it easier to evolve your products in line with future legislation.
4. You can build compliance into your digital products.
   Privacy by design is here to stay. If you already use service design, you can include customer information as a data layer in your service blueprints. If you don’t, now’s a great time to start. Mapping where data is collected, processed and stored highlights weak points where potential breaches may occur. Automated compliance tools will help lessen the burden on companies, plus has the potential to make data processing more secure.
5. GDPR supports innovation — if you do it right.
   Some warn that GDPR is suffocating innovation by restricting data flows and especially by deterring companies from innovating with data. Others point to opportunities to innovate with blockchain, IoT and AI in a way that’s secure and where data is protected. The truth? Yes, of course, you can innovate and be GDPR compliant. But ethics in AI is vital: you must respect your customers and their data.
6. Keep an eye on your third-party partners.
   This goes back to the joint controllers decision above. Companies now share responsibility for customer data with any third parties who process it and that processing must be documented. You can expect third-party checks, monitoring and contractual obligations to be a priority for companies from now on.

Here’s How GDPR Could Develop

Phew. That’s a lot to take in. But looking ahead, here’s where I’m betting we’ll see change.

1. GDPR will continue to evolve, with clarity coming from test cases and potentially further legislation including the ePrivacy Regulation.
2. The EU will continue to promote international adoption of data privacy law. We’ll see more countries embrace data protection, often baked into trade and security agreements.
3. If we’re lucky, we may start to see international convergence of data privacy legislation — especially if the US implements data privacy at the federal level.
4. But we’ll also see more clashes between the EU and the US, because of their opposite approaches to privacy.
5. As ‘data is the new oil’, we could see more situations where users receive free products and services by giving away data through cookies.
6. Businesses will shift away from third-party cookies and towards server-side tracking and automation, in order to stay compliant.
7. Businesses will adopt Privacy by Design (PdB) and service design tools and process, to help them stay compliant to multiple sets of privacy laws.
8. And finally — and this one’s a definite — we’ll see more and bigger privacy lawsuits. Who’ll emerge as the winners — big tech or privacy advocates? That I don’t know, but we can be certain of one thing: privacy lawyers will make a lot of money.

A Final Word On Trust

The theme underpinning both the European Commission’s communications and the commentary from industry experts is trust. Digital agencies like ours now need to provide evidence of data security and GDPR compliance — even down to staff training policies for data protection. That’s new. The EU’s priority is to support safe, secure data flows and innovation, both within the EU and outside. Standards compliance is their solution for this. And we, as designers and developers, have a crucial role to play.

• Part 1: GDPR, Key Updates And What They Mean
• Part 2: Cookie Consent For Designers And Developers

Other Resources

• Data Protection, the EU’s site
• UK ICO’s Guidance On Cookies
• GDPR Enforcement Tracker, logs fines applied under GDPR
• GDPR Checklist, by Cyber-Duck (a great place to start)
• Overview of Data Protection Law in the United States, by ICLG
• GDPR & CCPA Comparison Guide, by DataGuidance and the Future of Privacy Forum
• CCPA vs CPRA, from IAPP
• Security By Design (Amazon)
• How To Protect Your Users With The Privacy By Design Framework, Heather Burns, Smashing Magazine

Further Reading

• How GDPR Will Change The Way You Develop
• The Autofill Dark Pattern
• Fine-Grained Access Handling And Data Management With Row-Level Security
• The Potentially Dangerous Non-Accessibility Of Cookie Notices

(vf, il, mrn)