Zum Schutz der Datenbankintegrität muss die Datenbank-Software über ein vollständiges Transaktionssystem verfügen, welches dem ACID-Prinzip genügt.

(In order to assure data integrity, the database software must feature a full-blown transaction system which adheres to the ACID principle.)

As it is widely known, MySQLs transaction system, which was introduced only in version 4.0, does not satisfy the ACID criteria. Since the BSI also offers the BSI Grundschutzzertifizierung, which follows the guidelines laid out in the IT-Grundschutzhandbuch, this means that in order to obtain this certification, one cannot use MySQL — at least not as a database system.

As a side note, a number of German contract partners as well as numberous government agencies require their suppliers to have a BSI Grundschutzzertifizierung.

As a rather funny side note: according to a recent analysis by Alvar Freude, it is very likely that the new E-Petition site of the German parliament will fail not only the tests outlined in its own requirement specifications as well as accessibility guidelines, but also the very IT-Grundschutz certification.

The target is clear. Berthold Brecht suggested in his Radiotheroie that giving people the possibility to voice their opinion unrevised, universally, will inadvertedly start to develop an opinion of their own and start to escape state supervision. This is what happened on the Internet, but the state authorities are afraid of losing grip on the situation since it was not planned to track down individual members of such a discussion group.

Like Wau Holland said: «Denen ist Brechts Radiotheorie auf die Füsse gefallen, sie haben es nur noch nicht gemerkt.» (”Brecht's Radio theory came to haunt them, they just didn't notice it yet.“)

Since in the meanwhile, the radio theory has indeed been noticed, and the Internet is said to be full of terror (funnily in accordance with Brecht's theory), politicians as well as the ITU are monitoring the situation in China very closely. Why? Because China has managed to establish structures to control the information visible in their part of the Internet. This means that evil terrorism – such as asking for a democratic government – will be noticed and the responsibles can be thrown into prison.

This of course causes a lot of envy among our local politicians. The German CSU politician Uhl recently asked for Chinese-style restricted Internet access in Europe as well: ”If the Chinese can do it, we can do it as well. In that case I take pride in being authoritarian.“

And this is not just a single case from a random politician: the International Telephony Union (ITU) recently set up a working group for tracing back IPs. Evidently, the aim is no longer to convert China into a democracy, but rather to turn the rest of the world into a dictatorship.

Or maybe it's all just an olympic feverish dream?

Also improved is the SPAM filtering support. It is no longer necessary now to prefix rt-mailgate with procmail. Usability is also massively improved, the menu is now on the left again rather than on the top. Only submenus open on top. The annoying thing is though that in menus of a certain depth, the menu bar jumps between top and left for the same menu since the top bar only ever shows the topmost level.

People who are afraid of wearing glasses can now also configure font sizes at will.

So it is time to congratulate Best Practical to their new release, and to look forward to deploying the new PGP feature.

Which distributions were affected?

All distributions which pulled their OpenSSL changes directly from Debian. Those are namely:

Debian Etch and Lenny, Ubuntu/Kubuntu/Xubuntu and related, grml, Knoppix and all living customizations and Univention UCS 2.0. Other Linux distributions may also be affected.

Known not to be affected are: Fedora, Debian Sarge, NetBSD, OpenBSD, FreeBSD, DragonFlyBSD, MirBSD, Gentoo Linux, Univention UCS 1.x, Red Hat Enterprise Linux, OpenSuSE, SuSE Linux Enterprise, CentOS, pfSense, m0n0wall, Sun Solaris 10 and prior and OpenSolaris.

What exactly is the problem?

Due to a slightly misguided valgrind warning patch, the only “random” element used in key generation and other random number generation processes by Debian was the process ID. Since typical process IDs under Linux range from 0 to 65'535, there were only 65'536 possible different keys generated by the OpenSSL toolchain, also including SSH.

This means specificially that an attacker needs only 65'536 attempts to bruteforce a key generated by any Debian tool during this period of time. The impact of this depends on the usage of the key: for SSH user keys, it means that an attacker can impersonate the affected user and log in as the affected user to any system where the key is in the authorized_keys file. For keys used for certification and encryption, such as SSH host keys and SSL certificates, an attacker can impersonate the affected SSH or web server, and can potentially read currently running and recorded sessions, depending on the procedure used for session key establishment.

How can I figure out if my key was affected?

Debian and Ubuntu have released tools for key analysis which scan for patterns of the vulnerable keys by connecting to named hosts and looking into user's home directories for authorized_keys files which contain the patterns. An updated version of OpenSSH for Debian and Ubuntu now ships with a tool to automatically discover and refuse the vulnerable keys.

My key is affected – what should I do?

The first point is of course to immediately update the affected packages if you use a Debian derived system. Then, generate new SSH keys and replace them on all systems where your old SSH keys are located. Replace them as well on the servers of this nasty customer who left for the concurrence – imagine what would happen if he found out that you left a vulnerable SSH key on his host and that his host was compromitted by your negligence.

All affected OpenSSL certificates should also be revoked immediately. Generate new certificates and let them be signed and re-issued through your CA. Commercial CAs should let you reissue the certificate with the same Subject until the end of the certification period you paid up to. Please note that revokation is a critical step here, otherwise people might still impersonate your old certificate which might, after all, still be valid.

Then make sure your infrastructure was not taken over by botnets through an insecure SSH key. Check for rootkits as well while you're at it. If your log host is affected, tough luck.

How urgent is this? Will I have to act immediately?

Yes, this item requires your immediate attention as there are already botnets out there which search for accounts with vulnerable SSH keys. The question is not “Does someone care about me little Internet user?” — these bots are out to compromise hosts and to send SPAM and malware to other hosts. They don't care if you are an attractive target, they attack anything they can find and try to send SPAM with it.

I have put my securely generated private SSH user key onto a Debian system. Should I replace it?

Yes. On a Debian system, your private key was not safe during the last 2 years. The system may have been compromitted during that time, or someone may even only have been eavesdropping your communication and have gained knowledge about your SSH key. You should definitely consider it compromitted.

I have put my securely generated public SSH user key onto a Debian system. Should I replace it?

This depends. If your key is an RSA key, it is not compromitted simply by putting the public key onto a server and authenticating against it. The SSH 2.0 protocol, as described in RFCs 4252 and 4253, part of the token being signed as challenge by the user is the “session identifier”, which is a hash from the key exchange. This effectively prevents replay attacks of authentication processes done using a non-vulnerable SSH key, because the random material used as challenge is not only controlled by the vulnerable SSH host, but also by the non-vulnerable client. Thus, the data your SSH key has to sign as a challenge is not vulnerable to the weak PRNG of the SSH server, and thus cannot compromise your key.

This is however not true for DSA keys. DSA has a weakness when used in the Diffie-Hellmann key exchange process, rendering it basically uneffective. If the attacker gets hold of the random number used by the Debian SSH server in the key exchange process, this can be used to calculate the private DSA key from the public key with a complexity of 2¹⁶, being 65'536.

Summary

• Change any key pair generated using an affected version of the pseudo-random number generator. This applies both to the user and host SSH keys, and is of course also valid for certificates.
• If you have used a DSA key or certificate on a host affected by the vulnerability, it must be regenerated.
• Assume that all data read from and written to a vulnerable machine may be intercepted and/or tampered with, like if no crypto layer had been applied in the first place.
• RSA keys used to authenticate to vulnerable hosts are secure.

Acknowledgements

Special thanks for this goes to Steven M. Bellovin, who took the time to go through an analysis of this entire process with me and to clear up my misunderstandings about the OpenSSH challenge-response procedure.

This proves the urgency for all administrators to replace their SSH keys immediately, not only the host keys, but also the keys of all users.

Way back in May 2006, one of the Debian developers ran valgrind on OpenSSL in an attempt to make it more secure. Along the findings of valgrind was an uninitialized buffer named buf in the ssleay_rand_add function in openssl/crypto/rand/md_rand.c. The programmer simply commented out the MD_Update call which added the random data to the pool in order to fix the presumed flaw.

This blind patch was not exactly the correct thing to do. The data contained in buf was exactly the random pool initialization data, which was now no longer being added.

Apparently, the OpenSSL team also had its part in this game though. The Debian developer sent the patch upstream, and it was approved for debugging purposes by the OpenSSL team. Apparently, this was slightly misunderstood by the Debian developer, so he committed the now-defunct MD based PRNG into the Debian codebase.

According to the audit trail of the corresponding Debian bug, the Debian SSL team approved the patch and released a “fixed” package in May 2006.

The impact

As soon as the new OpenSSL release was deployed, the Debian users would now create keys using an MD as pseudo random number generator with hardly any modifications in the randon pool. As a short explanation to non-cryptographers: it was not really random.

The Debian Security team then discovered certain patterns which would emerge magically in most of their SSH and SSL keys, as well as keys from all other products which were based on OpenSSL. After several days if not weeks of analysis, the culprit had been tracked down to be that precise valgrind-triggered change.

The effect of this could be observed in the past couple of days by close followers of the Debian community. All of a sudden, the web certificates changed, all authorized_keys files were removed from the project servers, and some SSH host keys had changed, even though non of them had expired. This confused the Debian community very much, and was perceived as “A large security incident immediately ahead”.

With the release of the Debian Security Advisory today, this expectation was finally fulfilled, and the incident was indeed a major one: users were asked to regenerate all OpenSSL generated cryptographic keys since May 2006. A script was released to detect and warn about common patterns(!) in the various key files.

Lessons learned

There are certainly various lessons to be learned from this, both on the cryptographic, the programming and the practical side.

1. Don't blindly trust valgrind's output.
   This has been repeated over and over again. If valgrind finds a presumed flaw in your code, it does not necessarily mean it is really a flaw. It must be investigated very thoroughly by the programmer, and not patched away lightly just because it's there.
2. Cryptography may be counter intuitive to a programmer.
   I personally can't stop repeating this. What might appear as a runtime optimization to a programmer can indeed be a timing based information disclosure on the cryptographic level, and what might look like an uninitialized variable might actually not want to be zeroed out.
   This is also an argument against GnuTLS I keep repeating. Cryptography is not something which can be handled just like that by any good programmer. One needs at least a diploma in maths and programming plus be a very focused computer geek and close follower of the cryptographic community to even be able to touch cryptographic products successfully. This is the reason why I have major concerns with the GNU community rewriting an SSL implementation from scratch just because they do not like the OpenSSL license.
3. A diversification of infrastructures may be useful at times.
   This might be a bit counter-intuitive to those who followed the argument from the last paragraph, but the sole reason why the chain of trust did not break for the Debian team was that besides their working OpenSSL PKI, they also had a working, trusted and distributed GnuPG PKI. Thus, even though all OpenSSL keys were compromitted, the GnuPG keys could still be used to verify the origin of various security credentials and to verify that the new key material et cetera was indeed originating from the Debian project.

That said, I would like to proudly add that neither the NetBSD base nor the pkgsrc version of OpenSSL are affected by this bug.

Audit trail

• 22:20: Added more precise information on what keys and certificates changed
• 23:25: Added reference to what exactly happened to get the patch approved

Nowadays, most Linux distributions use the Novell build of OpenOffice.org for historic reasons. Originally, the OpenOffice.org community did not have the capabilities to offer an up-to-date build in a timely manner, so everyone fell back to the Novell build, which was usually very much up to date.

Nowadays, OpenOffice.org provides its own, very useful and stable build. However, most distributions did not adapt their packaging tools to this fact so far, and are still distributing the Novell build. Which – as it turns out – has some interesting fallacities. These are mostly bugs or build environment incompatibilities which cause the experience of running OpenOffice.org to be significantly impaired.

It is of course nevertheless a good question whether or not this is caused by a policy on behalf of Novell, or pure negligence of their build environment. Either way, it does not show a high value associated with the Open Source Community on behalf of Novell.

History

In the past, speculations about the questionable usefulness of Novell to the Open Source community had already been raised – especially in the question of the Mono-ization of Gnome. Also, concerns had been raised about partiality in the debate about OOXML, which is closely related to the Office debate since OpenOffice.org is the most popular implementor of the Open Document Format (ISO 26300) and as such in direct competition to the MS-Office-only newcomer standard OOXML.

Unfortunately, this also means that all links to the domain are now broken. More than that, the author originally asked for contributions to a new revision (in accordance with FFII's recommendation. Evidently, all these contributions are now lost along with the site.

Rest in pieces, no-dmca.ch!

The time for these measures appears particularly badly chosen, especially in the light of the fact that the big producers, such as Sony BMG or Warner Music, have recently decided that there is no added value in the use of DRM in copyright protection, and no longer DRM protect their media. Under these circumstances, the entire process should be questioned again before pointless measures are taken.

Investigating the situation in SwissIX, there were various incarnations of always the same packets flooding the network – it looked entirely like a switch loop. In theory, however, all switches used in datacenters should do spanning tree in order to avoid this situation, so this explanation appears somewhat vague. A mail on layerone-customer however also claims that a switch loop in InterXion was causing the failure. Maybe an unusual situation triggered a bug, causing STP to fail?

Cyberlink restored connectivity pretty quickly, once the interfaces at SwissIX were shut down, the traffic ran over Germany and similar fallback routes. swissix.ch remained unreachable, also via Swisscom GPRS. The SwiNOG coordination network was split into two pieces, rendering it ad absurdum.

Solnet took more than 2 hours to recover. Around 15:30, the full packet loss after the main Zuchwil router converted to a routing loop with the nexthop. Around 16:30, connectivity started to get back to normal. Maybe there are too few non-SwissIX peers available for this case?

Unfortunately, the outage falls right between the moving of the NGAS.ch network monitoring infrastructure, so there are no concise graphs available showing what exactly had happened. This has been corrected immediately afterwards.