strobe.cc - new articles

CUDA atomics: a practical analysis (updated)

2009-12-30T00:00:00Z

Contents

Hardware and assumptions
Precision and accuracy
What does clocks() return?
How long does an atomic operation stall a thread?
- Benchmark
- Analysis
Memory access patterns with delays between accesses
- Benchmark
- Analysis
Do adjacent memory operations cause atomic collisions?
- Benchmark
- Analysis
To be continued…
Acknowledgments

NVIDIA’s chips can have huge numbers of threads in-flight at a time; on my GTX 275, nearly 30,000 threads can be in the midst of executing. There is limited thread synchronization between threads on the same processor, and no inherent synchronization between other processors. Any coordination between threads must be achieved by writing to global memory, an activity with a large latency penalty.

As such, the atomic operations in the CUDA ISA are critical to some algorithms. In typical fashion, NVIDIA provides little guidance as to the performance of these operations, or as to the manner in which they are implemented. On a traditional CPU with a reasonable number of cores, one might simply suggest a few benchmarks, and indeed some benchmarks have been done on GPUs. The issue is that those benchmarks reveal that, for an operation as simple as an addition of a single variable, performance of atomic operations can be thousands of times slower than a traditional read-modify-write cycle. Under the right circumstances, they can also perform faster than non-atomic instructions.

For those crafting algorithms which use a number of atomic operations, understanding what causes the enormous difference in performance operations makes it easier to create optimized implementations. Along with a few others, I’m working on one such algorithm. This article will attempt to use benchmarks to help uncover the architecture used for memory operations in CUDA, in order to make writing such algorithms less of a trial and error affair.

I’m aiming to build an understanding of the architecture by asking a testable question, benchmarking, crafting a testable hypothesis as to why the results are the way they are, and then repeating the cycle until we’re out of surprises. It’s possible that this will lead to bad predictions (in fact, it already has); for the sake of conciseness, I’ll edit the incorrect conclusions out (and possibly place them in a scrap-heap article so that everyone who wishes to can still mock me for being so very wrong) rather than describe my backtracking.

Hardware and assumptions

These benchmarks are being done on an NVIDIA GTX 275 GPU, running at standard clocks, plugged into a Intel G965 motherboard, driven by a Core 2 Duo 6400 with 6GB RAM. The fact that my motherboard only supports PCIe 1.0 shouldn’t cause a difference with any of these benchmarks, as they exclude any host-to-device latencies.

My understanding of NVIDIA’s GT200 architecture comes largely from NVIDIA’s own documentation and the analyses done by Real World Tech and Beyond3D. I suggest the latter two if you’re not familiar with the architecture.

It should also be noted that my background is in audio and video, not in 3D, so this will be focused exclusively on general-purpose computation. While certain tricks may exist that use dedicated hardware for better performance, if you can’t do it from CUDA, I don’t consider it here. That might be a stupid move, but we’ll see.

These benchmarks are being conducted using PyCUDA, with data processing handled by numpy and scipy. They’re all written in PTX, NVIDIA’s own assembly language¹. Output is rendered by matplotlib. The source (caution: ugly) is available here.

1 PTX is a rather nice assembly language, and in some ways is still rather high-level. After trying both CUDA and PTX, I find that PTX allows me to write optimized code more easily, and involves fewer guesses about what’s going on than CUDA’s C compiler.

Precision and accuracy

Oh, and before we get started: you should probably know that my last attempt contained flagrantly incorrect results. So maybe don’t trust these so much until the article is done, okay?

For the curious, or those prone to schadenfreude: the benchmarks previously presented here were entirely correct; they accurately reported the number of clocks it took to run the given kernel. They seemed exceptionally strange, and they were—how many architectures do you know of where an atomic operation beats an unsynchronized one?—but I verified all the data being written by the kernels, ran the tests dozens of times, reread the statistics and graph code to make sure that everything was right. It all checked out. In fact, it checked out exceptionally well; most runs ended up with averages that stayed within one cycle of each other, and several would produce exactly identical results across each warp in each SM.

I figured that with so much consistency on my side, I had to be right. I even came up with an explanation which was consistent with all public information about the chip, one that seemed to satisfy myself and others working on the same project. Sure, the numbers implied a sustained write performance of 7 TB/s to global memory, but that’s not unreasonable for a large, wide cache.

Ultimately, I uncovered my error when testing my explanation. The test kernel used a single 32-thread warp to probe memory locations in a pattern designed to uncover the total cache size and cache line size of the supposed writeback cache. The results were fantastical, absurd; either NVIDIA hid more than a billion extra transistors on the die for a 32 MB SRAM cache, or something was wrong with my technique.

I had fallen into a trap which has ensnared scientists and engineers for centuries: assuming that accuracy meant precision. When an experiment contains a systematic error, sometimes the data is kind enough to be inexplicable, at which point most rational people will go back and recheck their experimental design. Unfortunately, humans are very good at explaining things. Given data which is remarkably consistent, we excel at conjuring a mechanism which explains it.

I suppose the moral of this story is that you should never use a black box to explain the behavior of a system unless you can set up an experiment to isolate and characterize the black box directly².

2 It is a very particular moral.

As for the bug, it was pretty trivial: I was repeating a short benchmark many times, and I assumed that the precision of the results indicated that they were an accurate representation of longer kernels. I also did not depend on the return value of the atomic operations. The out-of-order capabilities of the compiler and/or chip were allowing all timing code to execute before any results were returned. The rest of the article will take this into account, and (upon completion) will also explore these capabilities directly.

What does clocks() return?

This one has nothing to do with atomics, but a good understanding is necessary for benchmarking. It seems like it could be a stupid question, as the documentation says it quite clearly:

"When executed in device code, returns the value of a per-multiprocessor counter that is incremented every clock cycle."

Okay, neat. Except, wait, which clock? One would assume that this refers to the frontend clock, which ticks twice for each warp, but does that leave the two half-warps with different clocks? Or does it refer to the clock on the backend, which ticks four times per warp, leaving us with up to four different values per warp?

The heart of this experiment is in these three lines:

    mov.u32     clka,   %clock;
    mov.u32     clkb,   %clock;
    sub.u32     clka,   clka,   clkb;

Register ‘clka’ ends up holding the difference between two samples of the clock. Running the kernel in a single thread per SM and dumping the results to memory, we get to see this value. Turns out it’s exactly 28 clocks, without deviation.

Running it at 32 threads per SM, the results stay steady at 28 clocks, and all results in a warp are equal, indicating that this is the frontend clock latched at the start of a two-clock warp. Setting up a tight 256-round loop and storing the sum of differences to memory, we find this result:

The uncanny exactness of 28 clocks per round disappears when you have more than more than 2 warps per SM. This makes a lot of sense; at 4 warps, with two cycles per warp instruction and two instructions per clock, a round-robin scheduler would take 32 cycles to come back to the first warp, giving enough room to hide whatever caused the 28-clock minimum latency. Adding a few instructions in between those operations suggests that each SM is pipelined to give that massive register file time to breathe (obvious), that the exactness of the 28 clocks may be related to accessing special registers like %clock (less obvious), and that register dependencies are caught and handled by the instruction scheduler (obvious in hindsight).

The tightness of the error bars, even as the card climbs past full occupancy, is misleading, as this is the mean of 256 runs per thread. Cutting down the number of runs per thread to 8 shows much less determinism in saturated SM scheduling, although it’s comforting to note that the algorithm in use tends to keep threads at approximately the same instruction count (in the absence of memory operations) without the explicit use of thread synchronization over longer runs.

Conclusion: clocks() returns the frontend clock at the start of a warp’s execution. On an underutilized SM which can’t hide instruction latency, the comparison adds 28 cycles of latency on top of whatever was between the calls; this drops to 2 cycles on a fully utilized SM. It should be safe to use clocks() for benchmarking.

How long does an atomic operation stall a thread?

Benchmark

For this question, we’ll consider five types of operations: ‘load’ and ‘store’, neither of which is sufficient to compare to an atomic operation like ‘add’ but are included for reference; ‘load_store’, the traditional read-modify-write approach to addition; ‘red’, which performs an atomic reduction—that is, it computes and stores to global memory, but does not use the value returned from the memory controller in subsequent operations³; and ‘atomic’, which explicitly uses the result.

3 In C/C++, the compiler should emit a ‘red’ automatically when you ignore the return value of AtomicAdd() and friends.

These global memory operations will be run in a tight loop with code that times each operation. For ‘load’, ‘load_store’, and ‘atomic’, an explicit register dependency is created on the return value of the global memory operation by xor’ing it with ‘clka’ in the example above before reading in ‘clkb’. This trick seems to prevent an SM from reordering the clock sampling to improve accuracy. It does not affect ‘store’ or ‘red’ operations, so the reported numbers there may be incorrect or at least misrepresentative. More on this later.

Three memory access patterns will be tested. The first goes straight for the jugular: all writes across an SM go to the same address, ensuring that all atomic operations cause a conflict. Each SM gets its own address, though, because having all processors write to the same location caused several system crashes during testing. This is expected to be nearly the worst case for atomic operations, and the results do not disappoint:

Ick. Let’s not do that again.

The next access pattern is less pessimal; each memory location is separated by 128 bytes, and each thread gets its own memory location, ensuring that no conflicts occur but also preventing the chip from coalescing any memory operations.

Well, that’s… tolerable. It remains to be seen whether atomics can be used for scatters in computation threads, but this looks like it wouldn’t cause too much damage. One last access pattern: this time, all threads are neatly coalesced, each accessing a 4-byte memory location in order, such that a warp hits a single 256-byte-wide, 256-byte-aligned region of memory.

Crap. That’s quite a bit worse. Sure, the total latency for an atomic operation is better, but the ratio between an uncoalesced atomic and read-modify-write latency is much smaller than that for the coalesced pattern, so the relative cost of atomic operations in this context is much worse.

Analysis

Take a look at the error bars in the above graphs. For the ‘all conflicts’ access pattern, there’s an enormous variability in the time it takes to serve requests; whatever mechanism is being used to deal with conflicting atomic operations isn’t capable of FIFO scheduling all of them. In the ‘uncoalesced’ access pattern, the error bars shrink substantially; the variability of the times it takes to issue the memory request is very low. Coalesced memory accesses also have very steady times for both the load and load-store operations, but have a higher variance for store, atomic, and reduction operations. Note also that coalesced reductions, which should in theory allow the scheduler more freedom to hide memory latency, take longer and have more variance than atomics which prevent a kernel from processing the next instruction.

To explain this behavior, we need a detailed model of the memory architecture of the chip. From the descriptions at Real World Tech and Beyond3D, along with a little inference and a few patent searches, we have some a priori knowledge. Stream Multiprocessors have independent computation hardware, register files, and shared memory, but they’re not entirely independent. Each SM is bundled with two others into a Thread Processing Cluster, which handles instruction fetch, scheduling, and dispatch, as well as global memory operations (including ROP and texture fetch). The TPC’s controlling logic (frontend) is in a different clock domain from the ALU, FPU, and SFU (backend), with the former at half the speed of the latter. The TPC is also connected to a crossbar bus that connects to the other TPCs and the memory controller, among other things.

US Patent Application 12/327,626 vaguely describes a GPU memory controller. Given the filing date and subject matter, it probably covers technology developed for Fermi, but Fermi and GT200 are not so dissimilar as to make the filing irrelevant. It states,

"In one embodiment, memory hub bus 240 is a high-speed bus, such as a bus communicating data and memory requests in data packets (a "packetized" bus). For example, high-speed I/O buses may be implemented using a low voltage differential signal technique and interface logic to support a packet protocol to transmit and receive data as data packets."

By indulging in some speculation, it is easy to envision a vague protocol for issuing memory transactions on this bus. For the sake of having something to test, even if it is later found incorrect, let us assume that each TPC has a finite queue for pending memory operations, and that the memory controller also has such a queue. A TPC issuing a memory transaction would queue it, mark some registers as dirty on the scoreboard, and post the request on the bus. Then—and this part is entirely speculation, as other mechanisms for doing QoS or rate-limiting are widely employed in buses like PCI-E and HyperTransport—the TPC waits for an acknowledgment from the memory controller indicating that the memory request was successfully queued. In the event that the memory controller’s queue is full, the controller would bounce a "retry later" message to the TPC. All of this is done over the packet-oriented bus described above. Atomic calculations are handled by a dedicated SIMD ALU on or near the memory controller.

This mechanism will be tested and refined as we go, but for now it does manage to account for a few of the curiosities in the first round of benchmark results. If we assume the proposed system is true, then:

The small but nonzero wait time of "set-and-forget" operations such as ‘store’ and ‘red’ under low-utilization conditions is the round-trip time to the controller. (When the controller’s not flooded, a ‘red’ performs more or less just as fast as a ‘store’, as we’ll see later.)
The increasing wait time and variance of ‘store’ and ‘red’ as compared to their typically-slower analogs ‘load’ and ‘atomic’, respectively, under conditions when the controller was starved for DRAM bandwidth—viz, coalesced, 32 warps/SM—are related to increased numbers of memory controller "retry later" rejection messages. In other words, the limited TPC memory transaction queue is filled by ‘load’ or ‘atomic’ instructions waiting to return, acting as an implicit rate-control, whereas the TPCs simply retry continuously when attempting to push a ‘store’ or ‘red’ at the GPU, and the loop of rejection packets floods the internal bus bandwidth (or packet rate limit) as well as the DRAM bandwidth, causing the slight penalty seen in those instructions on the latter benchmark.
The limiting factor causing the decrease in the ratio ‘load_store’/’atomic’ in the coalesced case is the memory controller’s ALU.

However, the explanation is not perfect, or at least not complete; it doesn’t seem to explain why uncoalesced operations have such a tight variance, nor does it answer any questions about how conflicts are handled. It also doesn’t include hard numbers, such as the width of the SIMD ALU at the memory controller or the depth of the transaction queues. But there are plenty of benchmarks left to run which could help clear up these matters.

Memory access patterns with delays between accesses

Benchmark

The same three benchmarks as above, but with 50 32-bit multiply-adds thrown in. Remember, on GT200, a multiply-add is implemented as four separate instructions, so this is actually 200 instructions or 400 front-end cycles of computation added in addition to the memory operation and loop construct.

Yes, atomic collisions suck. But we knew that.

Note the performance of memory operations when the memory core is underutilized. Promising.

That’s right: free atomics.

Analysis

If you have the luxury of using coalesced memory operations, the performance cost of atomic operations which use the result are essentially identical to that of a read-modify-write cycle. The performance cost of a coalesced ‘red’ operation actually beats ‘load_store’ handily. If your kernel has enough number-crunching instructions between memory accesses, then the performance difference between any of these is insignificant, as long as the memory controller is not flooded⁴.

4 Coalescing uses the memory controller more efficiently, so it reduces the load, but the same effect can be achieved for uncoalesced memory writes if your kernels perform more computations between writes, as we’ll see in later benchmarks.

The particular conditions determining when atomic operations are "free" depend on a number of factors, including kernel length, SM occupancy, register dependencies, and memory access patterns. For example, the test kernel’s ‘filler’ instructions all depend on the result of the previous instruction, so it takes an occupancy of 8 warps/SM to hide register file latency and fully utilize the ALU. A different kernel might be able to swap threads more frequently, meaning that 1/8 occupancy might fully load the ALU. Of course, such a kernel might also issue more memory transactions as a result of its faster rate of execution, which could lead to the bandwidth constraints that result in higher penalties for memory operations. In other words, if you absolutely need atomics to be "free", benchmark your particular code!

On the other hand, these results also show that it’s not hard to get free or at least cheap atomics. I had prepared a complex workaround for the flame algorithm to avoid using these "slow" operations, and the flam4 implementation just gives the finger to atomicity and doesn’t attempt to avoid collisions (granted, they shouldn’t be that common, but still). Both of these tradeoffs were intended to avoid the high perceived cost of atomic operations; neither, as it turns out, were necessary.

These results are consistent with the proposed model for operation of the memory controller, but do not provide significant refinements to that model. The delays in the uncoalesced access pattern provides a bit more support for the theory that atomic operations are handled on-chip by a SIMD ALU; presumably, the flood of single-location uncoalesced memory requests were causing the ALU to be saturated with 1-vector operations.

Do adjacent memory operations cause atomic collisions?

Benchmark

Each CTA is given its own 32K region of global memory. The first eight lanes of each warp in a 32×8 CTA choose a memory address, so that each is offset from a 4K boundary by the distance under test. The result is that each warp places 8 memory accesses per iteration, each exactly 4K apart, and each offset from a 4K boundary by the same distance per warp, but a linearly varying distance across the CTA. It’s easier to understand with an equation:

address = 32768*ctaid.x + 4096*ctaid.x + OFFSET*ctaid.y;

Note that ‘x’ and ‘y’ are in the opposite order from what you might expect, to prevent memory accesses from being coalesced. A concern with this method is the potential to exhaust the number of queued memory operations per local TPC scheduler; this motivated the choice to limit to 8 memory operations per warp, which may help to avoid that condition, but will be investigated again after the queue depth tests.

We expect to see a penalty for atomic operations with a very high peak at low offsets, which drops off sharply to draw nearly even with a load/store operation at higher offsets. We are not disappointed:

Analysis

CUDA is geared towards parallel computation, and the memory architecture benefits from coalescing memory operations. Given the bias towards implementing memory operations with large widths, regardless of the actual amount of data requested, it seems likely that the mechanism which prevents atomic transactions from interfering with each other also operates on something more than a byte at a time. This, really, is the ultimate purpose of asking this question; it can be imagined that if you’re relying on atomic scatters to make your algorithm feasible, you’re not likely to implement a complicated mechanism for preventing adjacent writes, so the information gathered here is simply being used to test and expand the model for GT200b’s operation.

Unfortunately, the data isn’t as much of a slam dunk as would be desired for further asserting the nature of the underlying architecture. The chip was running a 30-block grid, which should allocate one 256-thread warp per SM, and yet the lowest time to complete an iteration is much higher for this benchmark than for previous benchmarks at this occupancy. Perhaps this discrepancy will be explained by later benchmarks, but for now have caution in interpreting these results.

While the scale may be different than what was expected, however, the relative sizes of the results are right in line with expectations. This benchmark is somewhat more probabilistic in nature than the previous benchmarks: not only is the thread execution order nondeterministic (as far as we know), the results of a collision can only be seen when that collision actually happens, which requires both memory transactions to be in the collision detection mechanism at the same time. For the all-conflicts case, assuming that both the memory controller and TPC have queues of sufficient depth to dispatch more than one warp’s worth of transactions at a time (8 transactions, in this case), a collision should happen for every thread iteration; as this is a deterministic result, we can arrive at conclusions by comparing the other results with the 0-byte offset case.

The results for a 4-byte offset are clearly very similar to those obtained for a 0-byte offset. With 8 different warps each writing 4 bytes of data with a 4-byte offset, every write happens within a 32-byte range per lane. Since the results are so closely to the collision-guaranteed 0-byte case, the data suggests that atomic writes within 32 bytes of one another are considered conflicting by the memory controller; in other words, we can say with confidence that the atomic lock width is at least 32 bytes.

At 8 bytes, corresponding to a 64-byte region of memory per lane, a run takes about 80% of the time per iteration as the baseline. Barring DDR shenanigans, we can chalk this improvement up to consecutive atomic operations that do not result in collisions. The mere presence of an improvement alone is compelling evidence which points to an atomic lock size of exactly 32 bytes, as a larger lock would have given the same performance for an 8-byte offset as for a 4-byte one. The trend continues, with a 16-byte offset having an even lower performance penalty, and the 32-byte offset having no significant performance penalty. As a whole, the evidence is strong for a 32-byte lock width.

An interesting and unexpected result came up when running this benchmark with more than one warp per SM. Take a look:

The performance boost at 64 bytes, and the further improvement at 128 bytes, points toward an increase in memory bandwidth when transactions are spread out past a certain width. Curiously, the 256-byte offsets and above aren’t quite as fast as the 128-byte case, although they’re certainly faster than the 32-byte offsets. (Again, these numbers refer to the offsets only; all of the actual write widths are 4 bytes.) These figures are probably due to the effect of the memory itself shining through the memory controller. They may be related to the way bank interleaving is done on the card, or to latency penalties for certain addressing operations. I’d have to study DDR’s operation a little more to draw conclusions from these results.

The first hints of the manner in which memory operations are scheduled and retired are also visible in these results, and once a few experiments are run which target the TPC and memory controller queue depths explicitly we can come back and validate these results, but the probabilistic nature of these results (along with the lack of nice assumptions like purely-random scheduling) makes extracting that kind of information a bit of a stretch, so I’m also setting that one aside for later.

To be continued…

This article is split into multiple parts, some of which have yet to be written. Questions that I’m working on answering include:

How many instructions are needed to hide atomic latency?
Does an atomic collision interfere with non-colliding transactions?
What’s the width of the memory controller SIMD ALU?
What is the depth of the transaction queue on a TPC? On the memory controller?
Does the hardware do any kind of out-of-order execution? If so, what?
Can instruction reordering by the programmer result in significant speedups?

Note: as of March 1, it’s looking highly unlikely that I’ll actually finish answering these questions before the release of GF100-based chips, after which interest (both yours and mine) is expected to wane considerably. Rest assured, I’m hard at work, just not on this.

Acknowledgments

My thanks go out to Sylvain Collange and Christian Buchner for valuable feedback and pointers at the CUDA forums.

Quodlibot, a Google Code IRC bot

2009-12-26T00:00:00Z

I was looking for a simple IRC bot which would announce changes to the Quod Libet project, like what CIA does for version control. Finding none, I hacked one together in about an hour, using Twisted and Feed Parser. It’s trivial, but code that isn’t shared is lost, and it may save a few others some time in the future. If you maintain a GC project and want me to host an instance of the bot, send me an email.

quodlibot.py (download)

# Copyright (c) 2009 Steven Robertson.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License version 2 or
# later, as published by the Free Software Foundation.

NAME="Google_Code_RSS_IRC_Bridge_Bot"
VERSION="0.1"

from twisted.words.protocols import irc
from twisted.internet import reactor, protocol, task

import feedparser

import re
import sys
import urllib2

class AnnounceBot(irc.IRCClient):

    username = "%s-%s" % (NAME, VERSION)
    sourceURL = "http://strobe.cc/"

    # I am a terrible person.
    instance = None

    # Intentionally 'None' until we join a channel
    channel = None

    # Prevent flooding
    lineRate = 3

    def signedOn(self):
        self.join(self.factory.channel)
        AnnounceBot.instance = self

    def joined(self, channel):
        self.channel = self.factory.channel

    def left(self, channel):
        self.channel = None

    def trysay(self, msg):
        """Attempts to send the given message to the channel."""
        if self.channel:
            try:
                self.say(self.channel, msg)
                return True
            except: pass

class AnnounceBotFactory(protocol.ReconnectingClientFactory):
    protocol = AnnounceBot
    def __init__(self, channel):
        self.channel = channel

    def clientConnectionFailed(self, connector, reason):
        print "connection failed:", reason
        reactor.stop()

class FeedReader:
    _schema = 'http://code.google.com/feeds/p/%s/updates/basic'

    def __init__(self, project):
        self.project = project
        self.entries = {}

    def update(self):
        """Returns list of new items."""
        feed = feedparser.parse(self._schema % self.project)
        added = []
        for entry in feed['entries']:
            if entry['id'] not in self.entries:
                self.entries[entry['id']] = entry
                added.append(entry)
        return added

def strip_tags(value):
    return re.sub(r'<[^>]*?>', '', value)

def announce(feed):
    new = feed.update()
    for entry in new:
        msg = '%s: %s' % (strip_tags(entry['title']), entry['link'])
        if AnnounceBot.instance:
            AnnounceBot.instance.trysay(msg.replace('\n', '').encode('utf-8'))

if __name__ == '__main__':
    # All per-project customizations should be done here

    AnnounceBot.nickname = 'quodlibot'
    fact = AnnounceBotFactory("#quodlibet")
    feed = FeedReader('quodlibet')
    reactor.connectTCP('irc.oftc.net', 6667, fact)

    # Don't reannounce every update on startup
    feed.update()

    update_task = task.LoopingCall(announce, feed)
    update_task.start(600, now=False)

    reactor.callLater(10, announce, feed)

    reactor.run()

This is your segfault on CUDA

2009-11-21T00:00:00Z

This is your segfault:

This is your segfault on CUDA:

Any questions?

The video codec formerly known as Fermi

2009-10-30T00:00:00Z

I’m writing a video codec!

To those unfamiliar with the video compression landscape, this seems like a bold and innovative move, and one which should generate much excitement from people who have that irrational love of squeezing their treasured collection of videos of their cats doing silly things in high definition down by an additional 2%¹. On the other hand, more experienced individuals might simply mutter, "another one?"

1 I’m one of ‘em. (The 2% part, not the cat videos. I do not have cats.)

Yes, folks, another stab at video compression, a field which generates hundreds of published papers a year (and likely many more unpublishable ones) describing how to eke out another 0.02dB PSNR from a Playboy centerfold snapshot or a ten-second CIF-sized video of some UT idiot fumbling a football². Hooray, you’ve saved five bytes on one video and increased decoding time by 600%. In MatLab.

2 Not kidding.

Okay, I admit it, that’s unfair. Engineering research is all about trying new things, even if they sometimes kinda suck, and publishing those papers may eventually lead to better real-world codecs. I can understand and almost forgive people trying to make tenure by trying ten ideas in a MatLab script and spacing out twenty 4-page papers over two years describing them. It should also be said that video compression is one of those fields in computer science where things get harder as time goes on: because compression is fighting against entropy, each step forward takes us another step towards a hard, nature-of-the-universe law, making it that much harder to attain the next round of stunning performance enhancements.

Of course, sometimes a new idea comes along which changes something fundamental about the way we do things and enables that next round of gains. For a couple years, a class of domain transforms which can be accurately but somewhat enigmatically be referred to as "directional multiresolution decompositions" have seemed like they could be that breakthrough in image and video compression. I’m preparing a blog post describing that… very impressive-sounding concept in high-level terms³, but for now it will suffice to say that these methods are new, promising, and just different enough to require throwing out most of the old tricks we used to squish video before.

3 After describing it to friends and family for a few months now, I might be able to pull this off in a reasonably articulate way.

As far as my preliminary paper-trawl has turned up, nobody’s even tried a directional decomposition video codec before, much less constructed one with a real-world implementation that makes impressive gains in coding efficiency. This either means that the topic is ripe for the researching, or that people have tried and failed abjectly. It certainly means that the topic is challenging.

Naturally, this means I pretty much have to try it, because I’m An Idiot™.

So, in one way, this is slightly different from most of the instances in which someone announces that they’re making a new video codec before they have code to prove it, as the fundamental underpinnings of the video codec are substantially different from any previous attempts I have seen and thus have the potential for producing significant and informative results.

In another way, it’s exactly like most of those instances, because by creating a new codec I’m ensuring that it will be at least ten years or so before it sees widespread adoption and general usefulness, even if it is a staggeringly good improvement. Perhaps it will take even longer than that. But, alas, there’s little room for avoiding that fate, as these changes can’t simply be patched into the Dirac or x264 code.

As I move from researching this problem to actually coding it, I hope to collaborate closely with existing open-source video codecs, -stealing- sharing code whereever possible and submitting patches enabling any new backwards-compatible compression techniques I might be so fortunate as to discover.

More as I learn it.

About the title

Oh, and I should mention: the codec was to be called "Fermi". Enrico Fermi derived Fermi-Dirac statistics independently of Paul Dirac, and since I plan to make use of as much code (including a significant amount of the bitstream specification) from Dirac as possible, I thought the name was apt. Except NVIDIA decided to steal my name for their latest GPU architecture. The product is still months away from launch, but Googling for "fermi video" already fills the page with noise about the cards, so a new name is required but not yet chosen⁴.

4 I’m calling it "The Video Codec Formerly Known As Fermi" in my head. FCAF (eff-kaff) for short.

Dear Typekit...

2009-10-28T00:00:00Z

Update: Typekit has done one better, not only using Gecko browser detection but also using WOFF for Firefox 3.6 and up. Took a few months, but they got it done the right way.

Dear Typekit,

Your service is currently broken. The good news: it’s a one-line fix.

The JavaScript generated for each client includes the folowing regex against navigator.userAgent, intended to check whether or not the browser is compatible with @font-face and Typekit:

function(D){
    var C=D.match(/Firefox\/(\d+\.\d+)/);
    if(C){
        var B=C[1];
        return parseFloat(B)>=3.5
    }
}

The problem is that many Linux distributors can’t legally call their browser Mozilla Firefox. The Debian project is a notable example; they’ve rebranded their Firefox build "Iceweasel", and chosen similar names for other Mozilla software. To avoid this dispute, other distributions have taken to using the code-name for a particular Firefox build as the name—I’m currently posting this from a browser called "Shiretoko". This doesn’t even cover contexts in which the Gecko engine is embedded by other applications which fully support TypeKit, such as Mozilla SeaMonkey.

It would be unreasonable to check for every variant of the browser name in the JavaScript handed out by your application. Fortunately, Mozilla has made a provision allowing the right decision to be made, regardless of browser name. Here’s the value of navigator.userAgent from the official build of Firefox:

Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.3) Gecko/20091021 Firefox/3.5.3

Here’s what it looks like in my browser:

Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.3) Gecko/20091021 Shiretoko/3.5.3

Note the common string component rv:1.9.1.3; this identifies the Gecko release version. Since browsers based on the Gecko rendering engine get most of their characteristics from that engine, you can in almost all cases simply check the Gecko version instead of the Firefox version¹.

1 Ideally, you shouldn’t use the user-agent string at all to do browser detection; Mozilla says why, and what to do instead. But often user-agent checking is the pragmatic solution.

Doing so is really this simple:

function(D){
    var C=D.match(/rv:(\d+\.\d+).*Gecko\//);
    if(C){
        var B=C[1];
        return parseFloat(B)>=1.9
    }
}

Since this change is a simple one, and since Gecko’s user-agent strings have been standardized for a while, I hope that you can make it soon, and enable support for the many users using browsers whch would otherwise work flawlessly with Typekit.

Thank you for providing an awesome and much-needed service.

Steven

PS: For users of unbranded Firefox versions who just want things to work now: go to about:config and change the string general.useragent.extra.firefox to Firefox/3.5.3 (or your current version). But remember to update it along with your browser.

Touchy-Feely: a TouchBook teardown

2009-09-23T00:00:00Z

The Always Innovating TouchBook is a touch-sensitive tablet/netbook device, powered by an TI OMAP-3530 processor and running Linux. They’re currently in very limited supply, although many pre-orders should be shipping soon, according to the company.

I was lucky enough to get one of the earlier models, and decided to tear it apart for everyone’s edification. The whole is, as usual, greater than the sum of the parts (viz, it works), but having a bunch of parts strewn around your house looks really impressive to non-technical friends.

There’s a Flickr set with more photos.

Uncasing it

I’m assuming that if you intend to tear this thing to pieces, you’ve already figured out how to get the back cover off. You’ll see something like this:

Before you go any further, pop off the battery’s tiny connector to the main PCB to avoid damaging the device or yourself, and pop off the speaker connector at the bottom of the board as well. The battery is glued to a metal shield, which is held on at four points by screws. You have two choices for removing it, and both kind of suck.

Choice 1: You can very carefully insert a putty knife or other thin, dull piece of metal under the top left corner of the battery, and slowly work it downwards, cutting the glue. Go slowly as you get towards the middle of the device. The glue stops, and if you go at it with the putty knife you might hit the small, delicate cable located below the bottom half of the battery (see below).

This will, of course, permanently weaken the stickiness of the glue; you’ll have to re-glue the device when you’re done with it. There’s a less-expected downside to this method: the battery is not reinforced, and is therefore pretty flexible. You will almost certainly bend the battery while doing this, and could damage or destroy it. (Remember, all of the exploding cell-phone/iPod incidents were caused by damaged batteries, and most of those are a lot smaller than this one.) Here’s mine after the removal:

Choice 2: remove the battery’s shield with the battery still attached to it. The shield is secured to the case at four points, as seen below (with the battery already unstuck via the first method).

When you’ve got all four screws off, very gently lift the battery towards the bottom of the unit, angling slightly upward and feeding the black cable through, as the cable is still attached to the display underneath with a very thin connector. When you have enough clearance, detach the display cable from underneath. This is what the cable looks like when attached:

Whether you want to break the glue or try not to break the cable is up to you. I went the first route, as I didn’t know what was underneath the battery when I started digging, but I may have tried the second method if I had.

Once the battery’s clear, using either method, you can unscrew the main PCB; two screws anchor it at the top of the unit, and one at the bottom. Rotate the board clockwise slightly, moving the top left away from the case; then lift the top of the board slowly until it is free. There’s one last cable anchoring the board to the case. It’s attached close to the left side of the PCB; you’ll need to rotate it like it’s hinged on the outside left of the case to see it without damaging it.

There’s no trick to this cable; just pull, gently but firmly, until it releases from the socket. (Reassembling works the same way, athough a pair of pliers may help in applying enough force to the cardboard backing the contacts to get it to properly mate in such a small space.)

The rest of the tear-down is pretty obvious. Take a look at the Flickr set linked above if you’d like to see the results.

Enjoy your TouchBooks!

Can androids render Electric Sheep?

2009-09-21T00:00:00Z

The Electric Sheep screensaver combines a genetic algorithm, an iterated function system, and some postprocessing to create one of the most mesmerizing visualizations I’ve ever seen. Over the summer of 2008, I looked into creating a version of the rendering library of Electric Sheep, a library known as flam3, which would use NVIDIA GPUs to accelerate their transforms.

The algorithm is pretty straightforward. In broad terms, a fractal flame, or simply sheep, consists of a set of equations, each having as input a coordinate and as output a coordinate . Start with a random point. Plug in this point’s coordinates into one of these randomly-chosen pairs of equations, and plot the result. Repeat. When it’s done, you get a fractal image. There’s more to it than that, of course - here’s Scott Draves’ paper on the subject if you want the details.

At first glance, this maps relatively well to the idea of a massively parallel processor like a GPU. Instead of running one point at a time, just run hundreds! Unfortunately, as you might have guessed, it ain’t so simple.

Part of the problem in getting the transforms over to the GPU had to do with NVIDIA’s CUDA SDK. Normally, CUDA is supposed to be written in a C-like language, which gets compiled to an assembly language equivalent. The assembly - written in NVIDIA’s own PTX language - then gets some register-allocation optimizations before winding up as machine code. The PTX emitted by the first implementation of the transform in CUDA’s C dialect did not agree with the register allocator, and the resulting code used more than 50 registers after allocation.

50 registers is really bad. In CUDA, each processor unit handles hundreds of threads, and attempts to hide latency by switching between threads rapidly. Context-switching has a low overhead because all registers are allocated at the start of a thread and remain allocated throughout the thread’s lifetime. In first-generation GPGPUs, maximizing the occupancy of each processor (and therefore hiding the most latency) required kernels which used a mere 10 registers. Needless to say, this code performed very poorly (for this and other reasons). After fighting with the C code for a while, I gave up on the compiler and decided to code the entire transform kernel in assembly.

It helps that PTX is actually a rather nice assembly language, as far as they go, but it still turned out to be a pretty significant task, taking me the better part of a season to even get to a workable state. Assembly can be challenging for large applications, but the time and effort had more to do with the unusual memory architecture of NVIDIA GPUs than the programming language in use.

Hardware interpolation

Some essential background information is necessary for the next sections to make sense; most of this is detailed in NVIDIA’s CUDA Programming Guide [pdf]. A lot of this section is well-founded speculation.

Computation on the GPU is handled by multiprocessors. Each multiprocessor on a current-gen (GTX 200 series) GPU is capable of tracking 1024 threads in-flight. These threads are grouped into warps of 32 threads; each of these warps has a single instruction pointer, meaning that the 32 threads must all execute the same instructions on different data. (Instructions can be predicated, meaning the results of those computations are not stored, so branches where some threads of the warp follow a different path can be simulated by following one path with some threads "turned off," then the other path with the complement threads disabled. Highly-divergent paths are an absolute disaster, performance-wise.)

Each multiprocessor will step through a warp in two "front-end" clock cycles. (The front-end faces the multiprocessor bus, which interfaces with the memory controller; the back-end contains the ALU and is double-clocked.) During each of these clock cycles, threads being executed can issue a memory access request. If the memory access meets certain criteria, the front-end bundles the requests into a single transaction and ships it off to the memory controller; if not, the requests are transmitted individually.

Main memory on an NVIDIA GPU is accessed by a controller that is shared by all processors. Memory requests are sent down a very wide bus to the controller, which queues them. The controller interleaves storage across all RAM on the board in order to increase bandwidth. When processing requests for contiguous blocks of data, this has a significant effect on performance; the latency cost of bringing 14 separate RAM chips (on my GPU) to the same address is less significant than being able to complete your transaction in a few clock cycles. Unfortunately, it means that the latency cost for accessing a single byte of data is at worst case 14 times higher than for non-interleaved memory architectures. I’m not sure if the controller can process requests for non-contiguous memory simultaneously if the regions are located on separate chips, but given the dire warnings against non-contiguous memory access scattered throughout CUDA documentation, I doubt it.

These latencies are compounded by the lack of any real cache. The decision to omit a cache from each multiprocessor is the right one; for most operations, a small cache would just miss anyway, a large one would be absurdly expensive. Worse still, keeping cache coherent between 30 processors would require jaw-dropping complexity. To sort of make up for it, NVIDIA places 16K of (usually) penalty-free RAM on each multiprocessor, which must be shared by all active threads. While it sounds like it could be used as a manually-managed cache, the size of the shared memory is too small to make that effective; at full capacity, you get a mere 16 bytes per thread. This memory is the only way to communicate with the other threads active on a particular multiprocessor, so it is more often dedicated to coordination instead of cache.

There is also a shared 8K cache per multiprocessor which shadows a region of memory only writable by the host system; this constant memory is useful in providing constants (transform coefficients, pointers to buffers, etc) to a kernel, but cannot be used for coordination as it is not guaranteed to be consistent if memory is changed by the host after the kernel is started.

These hardware oddities can be summed up in three simple design rules for CUDA applications:

Split your tasks into groups of 32 threads.
Make these groups branch as little as possible.
Access memory infrequently and contiguously.

Porting flam3

These three goals may seem trivial, but they can be Zen-like in their elusiveness. Some unexpectedly difficult parts of the port are outlined below, along with solutions, in no particular order. I’ll continue to update this as I learn more.

Random number generation

The iterated function system at the heart of the fractal flame algorithm depends heavily on high-quality random numbers. The current implementation uses the PRNG. However, a reasonable ISAAC implementation requires a minimum state of 144 bytes per random context, and ISAAC produces random numbers using a procedure that must operate in serial over the context. At a minimum, using atomic transactions to block the execution of any other thread on the chip, and keeping one random context per warp, the context alone would consume 4,608 bytes. This blows through the entire allocation of shared memory for a 256-thread warp without storing a shred of data about the IFS itself.

Stronger random number generators, such as the popular Mersenne Twister, could have been selected; these generators would have to be called separately to generate and store a large block of random numbers, which would then be read in as needed by each warp. This solution may be the best in terms of quality of generated numbers, but the coordination required to get this working was deemed too expensive for a first effort.

Ultimately, I decided to go with an MWC algorithm, using different values of a for each thread selected from a pregenerated table distributed with my build. Given the extensive list of compromises already made in porting flam3 to the GPU, I doubt that an MWC algorithm would make a significant impact on the results of the computation. This method was implemented using two persistent registers per thread, and required no shared memory. I will reconsider this decision after I see CUDA 3.0’s memory hierarchy, which arrives alongside the GTX 300 series sometime soon.

Transform data structure

In flam3, a single (x, y) pair of functions in the IFS is created from a series of fixed-form functions that operate on the input coordinate pair. The outputs of each of these are summed together with weights applied. The transform description includes the list of functions to run, information about the weights of the functions, and for many functions, coeffecients or other parameters. On the CPU, these transforms are stored as arrays; the coeffecients for every transform are present and simply ignored if the transform is not in the list. These arrays take around 8K per transform, and there can be many transforms in a particular sheep. Even using constant memory, this creates an unacceptable amount of memory usage and traffic for the embedded architecture.

Instead of storing the transforms in this format, the CPU reads the entire set of information about the transforms and pushes it onto a stack. The stream is then read in sequence when on the GPU; each transform is executed in order, and all information is popped from the bottom as the transform is processed. This cut most transforms to 200 bytes, allowing the entire set of transforms to fit in the constant memory cache and allowing multiple transforms to be run at one time. It’s obvious in retrospect, but it took me a while to see this strategy.

Trigonometric functions

Many processors don’t have trig primitives implemented as instructions. If you find yourself in an assembly language and need an arctangent, compute a Taylor series for the function of interest. Be careful to measure the divergence of the series from the function being modeled; for things like a tangent, you may have to clamp the input values to a certain range depending on the size and precision of your series.

Storing transform output

The brightness of fractal flame images is computed as the log of the density of the points in a sample region. Hence, the bright spots on an image correspond to a set of counters in memory which have been written to hundreds or even thousands of times during the course of a render. Combine a high write density to a particular region of memory, a massive thread count, and enormous delays between reading a memory location, adding to it, and writing the result, and you get one of two things: either a result which can be off by an order of magnitude, or a desperate need for atomic transactions.

A previous approach to this problem involved the latter: simply do every operation on the framebuffer using atomic intrinsics. Unfortunately, this slaughtered performance. Not only was the memory interface crippled by millions of separate I/O requests, the majority of them suddenly became atomic - stalling most of the execution units on the chips. I don’t have hard numbers for the performance hit yet, but theoretical caluclations showed that the code achieved less than 2% of its expected throughput with atomic writes enabled.

I’ve thought of a different approach - a considerably more complex one, to be sure, but also one that is much more suited to the GPU’s memory model, and will almost certainly yield great gains in practical rendering performance. It may even be possible to achieve real-time performance at near-HD resolutions on GTX 300 cards, opening the door to a new class of sheep visualizations taking their input from real-time data. It will take some time to implement it, but I plan to restart my work as a hobby project over the next few months, and get things ready to test when the new GPUs roll out.

The algorithm addresses the key aspects of the memory system: small working set, high latency, no externally-controlled consistency, and efficiency gains with contiguous requests. It is conceptually simple: instead of writing the results of the computation - a coordinate pair and a color value - directly to the frame buffer, pack them into a 32-bit int and write them into a log. When the log is full, hand it off to another thread. This thread will read the log and split it, moving the contents of the log into one of 32 smaller logs, each corresponding to 32 subdivisions of the image. Another thread will take these logs when they get full, further dividing them until each log corresponds to an image area of 256 pixels. 256 four-color pixels, at four bytes per pixel, gives a total memory size of 4,096 bytes. This is small enough to fit three 256-wide thread blocks onto a multiprocessor at a time, ensuring 75% occupancy of each GPU multiprocessor, giving the device enough threads to work effectively without stalling for memory.

Here’s the gotcha: there’s no central coordination mechanism on NVIDIA GPUs. While they run, each block of threads can communicate to other threads in the block of memory using shared memory, and with other blocks using main memory, and that’s it. So, while this simple strategy takes the memory delay from to , it also involves writing a memory allocator and threading library from scratch, in assembly, without a debugger and using nothing but a few atomic intrinsics. It’s not an impossible task by any means, but it is a significant challenge. I’ll have more details on this process as I go about it.

Moving forward

I’d like to finish this up sometime. I’ll be investing in a new system to coincide with the upcoming release of new graphics cards sometime in the future, and I’ll also be performing a lot of heavy lifting on GPUs and DSPs as I work on my thesis. I will investigate OpenCL as well; there’s no doubt it would be easier to code this thing in a high-level language¹.

1 God help me, I just called a C-based language "high-level".

I’ll keep you posted.