Implementing Header Bidding to Improve Network Advertising

NOTE: This is a mirrored tech blog that I published in Japanese at the company blog while I was working for Cookpad in 2018.

Advertising Development at Cookpad

In November 2015, an article titled "What Cookpad's Ad Engineers Do" was published, introducing the development activities of the advertising development department. However, more than two years have passed since then, and the situation around the ad delivery system has changed significantly. First, I'd like to give a brief overview of the current state of ad development at Cookpad.

The Media Product Development Department, where I belong, is responsible for developing both ad delivery systems and video streaming services. In the past, our team has published technical posts about video streaming technologies, such as:

• "Rushed Auto Scaling Environment Construction for cookpadTV Live Streaming Service"
• "Comment Delivery Technology for Cooking LIVE App cookpadTV"

The services we're responsible for in the ad delivery system development include:

• Development of ad delivery servers (Rails)
• Development of in-house ad submission systems (Rails)
• Operation of ad logging infrastructure (Python, Kinesis Streams, DynamoDB, Lambda)
• Development of SDKs for ad delivery (conforming to each platform; JavaScript for web)

In terms of projects, our scope includes:

• Improvement of existing in-house ad products
• Development of new ad products
• Development and improvement of network ad products

For example, from the end of last year to the beginning of this year, I was involved in the development of a new ad product, and slides about part of that project have been published. Please check out "Ad Delivery Server and Ad Delivery Ratio Optimization Problem."

In this post, I'll introduce one project in the "Development and Improvement of Network Ad Products" category.

Background

At Cookpad, we deliver two types of ads:

1. In-house ads
2. Network ads

For network ads, we deliver advertisements through multiple ad servers in partnership with various Supply Side Platform (SSP) companies, but these ads have been hitting a ceiling. Therefore, we needed to work on improving ad revenue.

Simply increasing the number of ad slots could lead to a decrease in user experience and an increase in network load, so we needed to avoid that. Consequently, we had to improve the purchase amount and delivery flow of currently served ads.

This led us to implement a mechanism called Header Bidding, which has been increasingly adopted in Japan in recent years.

About Header Bidding

Header Bidding is a process where you auction ad space to various SSPs for the best price before making a request to the ad server.

Mechanically, it's called "Header Bidding" because bidding requests (= Bidding) are made in advance within the tag (= Header).

Without Header Bidding

Let's look at the traditional pattern of ad space purchasing without going through Header Bidding. Here are the definitions of terms in the diagram:

Term	Description
Client	The side displaying ads, in this case, the main Cookpad site
Ad Server	Ad server
SSP	Various SSP companies
Floor	Floor price
Bid	Bidding result
Winning bid	Successfully purchased bid price

The bidding logic of existing ad servers basically follows a waterfall method for purchasing. Therefore, in the case shown in the diagram:

1. The floor price for the ad space is $1.0
2. SSP α's bid result is $0.8 (below the floor price)
3. SSP β's bid result is $1.2 (above the floor price)

As a result of querying in order from top to bottom, "SSP β"'s ad will be displayed as it was the first to bid at a price above the floor price.

Please note that "SSP δ"'s bid result is $2.0. If we could have reflected "SSP δ"'s bid result, the value of that ad space would be $2.0. In other words, the true value of the ad space is $2.0. However, due to the constraints of the waterfall method, the bid result of $1.2 was reflected (a missed opportunity of $0.8).

Header Bidding solves this problem.

With Header Bidding

Next, let's look at a case involving a bidding server. Note that I'll explain using the Server-to-Server method discussed later.

In this case, the process is executed in the following order:

Order	Flow	Description
1	Client -> Bid Server	Request Header Bidding to the bidding server
2	Bid Server -> SSP	Bidding. At this time, bids to each company are made simultaneously
3	Bid Server -> Client	Return the bidding results. Here, since "SSP δ" returned a response to purchase for $2.0, the Winning bid becomes $2.0
4	Client -> Ad Server	Request an ad after receiving the bidding results. By communicating the Winning bid, "SSP δ"'s ad will be returned

Unlike the case without Header Bidding, you can see that the bid result from the SSP with the highest purchase amount is reflected.

The key points are:

• Conducting bidding before requesting the ad server
• Processing requests to various SSPs in parallel

This allows us to optimize bid prices that would otherwise be lost.

Client vs S2S Header Bidding

There are two types of Header Bidding:

1. Client Header Bidding
2. Server-to-Server Header Bidding ** Client Header Bidding is a format where bidding is done on the client side. Technically, within a <script> tag, you specify a list of SSPs to bid to and send bid requests to each. It's a format where you wait for the results of those bid requests and select the bid result with the best eCPM.

Server-to-Server Header Bidding is a format where bidding is done on the server side. The difference from Client Header Bidding is that you only need to send one request to the bidding server. Also, the bidding server handles the bidding logic (e.g., waiting for bid results from various SSPs, timeout processing, comparison of bid results), significantly reducing the responsibilities on the client side.

Currently, the Server-to-Server method is mainstream.

Header Bidding Services

We've adopted Transparent Ad Marketplace (TAM), a Header Bidding advertising service provided by Amazon.

Design & Implementation

Basically, the implementation is complete once you embed a script that requests the bidding server in the <head> tag, following the documentation provided by TAM.

However, in our case, we needed modifications because we couldn't smoothly introduce it due to constraints such as:

• Delivering all ads, both in-house and network, through our own ad submission and delivery servers
• Ad slots delivered on each page are not static but change dynamically

Below is a diagram showing the overall flow of Header Bidding. Here:

System	Description
ads	Internal ad delivery server
display.js	JavaScript SDK for ad display
cookpad_ads-ruby	A simple gem that defines Rails helpers for embedding display.js
apstag	Header Bidding library provided by TAM
googletag	Ad network library provided by DFP

Note that official documentation for googletag can be found at here.

The general flow for executing Header Bidding is as follows:

1. The JavaScript SDK requests ads to be displayed from the ad delivery server
2. If the ads include network ads, the Header Bidding process begins
3. First, initialize apstag and googletag (e.g., default timeout settings)
4. Use apstag to send Header Bidding requests to TAM
5. Based on the bidding results, send ad requests to DFP
6. Once ad requests are returned from DFP, display the ads

The key point is that while requesting Header Bidding:

• Pause DFP requests with googletag.pubads().disableInitialLoad() to conduct Header Bidding
• Once bidding results return, resume the ad rendering flow with googletag.pubads().refresh([opt_slots...])

Results

With the above, I've explained the entire process of implementing Header Bidding. While I'll keep the specific numbers private, we were able to achieve improved revenue for network ads through this implementation.

New Challenges

Degradation of Ad Rendering Flow Performance

However, new challenges emerged here.

The issue is that the latency until network ads are displayed increased by the amount of Header Bidding requests, resulting in a performance degradation of roughly 150-400ms, which is quite critical.

Below is a diagram showing the sequence of rendering processes until an ad is displayed.

(Processing, DOMContentLoaded, and load are general terms for the sequence of browser steps to parse and render HTML/CSS. If you're interested, check out "Measuring the Critical Rendering Path" by Ilya Grigorik.)

What's immediately noticeable is that everything from the request to the internal ad delivery server (ads), to Header Bidding, to requests to DFP is executed serially. By introducing Header Bidding, latency increased by that amount.

Lack of Visualization for Ad Rendering Flow

I described it as "150-400ms" based on my personal feel, but the ad rendering flow in the client environment had never been measured or visualized before.

While we want to improve the performance of the ad rendering flow mentioned above, we can't know exactly where the bottleneck is until we measure it. As they say, "You can't improve what you can't measure," so we first introduced a flow for measurement and visualization.

Fortunately, we already had a mechanism to send logs to Fluentd and build tables in an analyzable data warehouse (currently Redshift for Cookpad). Therefore, we could achieve this by simply writing some table definitions on the client side.

Note: For more on Cookpad's data utilization infrastructure, see "Cookpad's Data Utilization Infrastructure."

Below is a visualization based on the acquired log data. Since we just started collecting logs recently, we haven't yet tackled the visualization flow. Our immediate goal is to create dashboards in the BI tool recommended internally for regular observation.

For the critical path of ad rendering, you can simply embed loggers at any timing, but for DFP, you can use googletag.events.SlotRenderEndedEvent to capture events when an ad space is displayed or when an ad becomes "Viewable."

Countermeasures and Future Outlook

Above is the explanation of countermeasures for issues that emerged from the introduction of Header Bidding. In the near future, we plan to work on the following:

1. Ad rendering flow visualization phase
2. Ad rendering flow optimization

For "Ad rendering flow optimization," we're currently in the design and PoC implementation stage with a policy to push forward the timing of requests to our in-house ad delivery server.

Specifically, we currently start the ad rendering flow at the bottom of the in the HTML file, but we need to improve this to start at the earliest possible stage in the tag (due to past design constraints, requests to the ad delivery server are blocked until each ad slot's HTMLElement element is inserted into the render tree and actually drawn).

I'd also like to introduce performance visualization and rendering flow optimization on another occasion.