Newcons coming to Debian GNU/kFreeBSD (testers wanted!)

FreeBSD vt(4) -commonly known as “Newcons”-, which is planned to replace Syscons as the default FreeBSD console, is now available on Debian GNU/kFreeBSD.  It is enabled by default, but only in kfreebsd-11 packages from experimental.

Newcons provides many interesting new features, such as KMS support, Unicode, double-width CJK characters, etc. More details are available in FreeBSD wiki.

Ironically, Debian GNU/kFreeBSD is in a bit more of a hurry to deploy Newcons than FreeBSD is. The reason for this is that Newcons is practically a requirement for using the new KMS drivers. While FreeBSD Ports, in order to ensure a smooth transition, preserve support for UMS (User Mode Setting) in their X11/DRI userland, Debian supports many kernels and (contrary to some ill claims I heard) gives priority to new features on Debian GNU/Linux port, which is the one with most users and developers. In this case, it means KMS is the only option when it comes to X11/DRI userland in Debian.

Anyway, we’ve been making some nice progress. Here’s a screenshot of Newcons in action in a Debian system (running on VirtualBox, thus with VGA backend):

The idea is to backport this to kfreebsd 10.0, as FreeBSD plans to merge it into stable/10 anyway. However, it is still in the process of being tested (and being polished on FreeBSD, I hear they plan to merge it on March).

Rooting the LG Optimus L3 (and just about any Android model, really)

I recently purchased an LG Optimus L3 (“e400”) mobile phone. I’ve to admit it would be a perfectly fine purchase if it wasn’t for the huge amount of time I had to invest in order for this thing to recognize me as its master. Even though I wanted to install CyanogenMod in it, the device blocked my attempts to install it through ROM Manager. Yes, for some awkward reason LG kept on thinking they still owned the device even after they sold it to me.

Anyway, as none of the rooting instructions for LG Optimus L3 I found on the net worked for me, I figured I should try with the recently-discovered “master key” vulnerability. A bit of research shows there’s very extensive documentation on this hole and how to exploit it, as well as proof-of-concept exploits. But unfortunately, I haven’t been able to find any complete solution to implement this in easy steps.

On the soulders of giants…

After running through the whole manual setup of this exploit (and succesfully rooting my LG Optimus L3), I figured it would be useful if I put the pieces together into a unified script.

So here it is, an exploit for the Android “master key” vulnerability which will give you root access to almost every Android device in existance:

https://github.com/robertmillan/mkbreak

Enjoy. Oh and btw: fuck you LG.

ZFS v28

ZFS v28 (finally, with deduplication) is available in latest D-I daily builds when selecting kernel of FreeBSD 9 as boot option.

Testers wanted!!

Debian GNU/kFreeBSD on production

Yesterday I begun using Debian GNU/kFreeBSD “squeeze” in thorin, my main workstation.

During the last few weeks I had to work through some of the limitations that were holding me back, such automated driver load and FUSE. I was lucky enough that other people filled the missing pieces I wanted, such as NFS client support and a GRUB bugfix that broke booting from Mirrored pools.

I have to say that I’m very satisfied. Barring a pair of small nuissances here and there, the result is quite impressive:

• System responsiveness has noticeably improved, specially on disk-intensive operations such as loading my browser. I believe lightweight ZFS compression (the default algorithm is fast enough that it saves time rather than spending it) has a big influence on this.
• Partition management became a lot simpler. Instead of using a fixed size like legacy file systems, with ZFS the storage is shared. This allows me to add, remove or resize file systems or swap volumes from my running system without having to worry about where they will be allocated (I have two pools, one that is mirrored and one that isn’t, and I can add or remove things from either. That’s as far as I’m concerned with allocation).
• I no longer have to go through long wait periods when adding a new physical disk, or replacing a broken one. I have a lot of old used disks at my disposal due to various reasons, and they tend to break easily, so I used to run badblocks to them before I begun using them, to be sure they were safe (reliability is very important to me). Now I simply plug them in and trust ZFS checksuming to do the right thing.
• No more waiting for long fsck. Specially annoying when I’m in a hurry and chance wanted that this was the 30th time I boot my system without fsck.

That’s basically my personal experience as newbie Debian GNU/kFreeBSD user. Of course my perspective is very limited because I just started and yes, I am biased.

Anyway, what about yours? If you have installed Debian GNU/kFreeBSD, was it meant for production or just a “toy machine”? If you considered using it on production, did it succeed at satisfying your needs, or did something hold you back? Leave your comment!

Recent improvements with Debian GNU/kFreeBSD

Debian GNU/kFreeBSD was first released with Squeeze in last february. The “technology preview” label indicated, among other things, that it had a number of limitations when compared with what users would expect: missing features, incomplete functionality, etc.

But it has seen many noteworthy improvements since then. Here are some that I would like to mention:

• Graphical installer support in D-I.
• Userland support for FreeBSD Jails has been imported.
• Userland support for encrypting disk partitions (geli) has been imported.
• FUSE support is now provided by fuse4bsd.
• After NFS daemons and utilities has been packaged, it is finally possible to mount NFS file systems.
• kFreeBSD (kernel of FreeBSD) has seen a number of updates:
  • Compiler has been upgraded to GCC 4.6.
  • New release (kernel of FreeBSD 8.2) now available in unstable and wheezy.
  • Regular snapshots of 9-CURRENT kernel are now provided in experimental. Among many other improvements, this brings in kernel side of ZFS deduplication.
• Wireless support, which was disabled in squeeze due to a bug in ifconfig, has now been enabled in sid and wheezy, then backported to squeeze after it was reported to be functional.
• After devd has been enabled in freebsd-utils, USB drivers and sound drivers are now autoloaded. In addition, allow-hotplug can now be used in /etc/network/interfaces.
• After some discussion with upstream, ABI compatibility with FreeBSD has been restored, and it is now possible to use Debian GNU/kFreeBSD in a chroot under FreeBSD.

Diskless Debian GNU/kFreeBSD HOWTO

It was completely non-obvious; In some cases I had to figure it out by reading kFreeBSD source code; I write it down here so it’s not forgotten ;-)

This is a short guide on how to boot Debian GNU/kFreeBSD via network on a diskless machine, using GRUB as the bootloader.

Step 1: Configure your DHCP/TFTP environment to load GRUB from network. See the PXE GRUB documentation for details.

Step 2: Rebuild kfreebsd-8 package with BOOTP and BOOTP_NFSROOT options, as described in upstream handbook. They need to be added to the debian/arch/*/*.config file for your architecture. This kernel can still be used for normal, disk-based boot, but it will issue DHCP requests to attempt network boot every time you start it.

Step 3: Put your kFreeBSD in TFTP directory and configure GRUB to load it. This requires only one command in grub.cfg. My setup:

set timeout=1
set default=0

menuentry “Debian GNU/kFreeBSD” {
echo Loading kFreeBSD …
kfreebsd /beastie/boot/kfreebsd-8.1-1-amd64.gz
echo Booting …
boot
}

Step 4: Build a directory tree with your GNU/kFreeBSD userland, and export it via NFS. I copied it from an existing Debian GNU/kFreeBSD install, but I suppose debootstrap could do the job as well.

Step 5: kFreeBSD mounts root NFS in readonly mode initially, and Debian userland turns out to be very unhappy with that. If you want things to work you’ll need to either setup tmpfs in a few places (e.g. /tmp) or adjust fstab so that / is remounted as writable by the INIT scripts. Example fstab line:

192.168.1.3:/srv/netboot/beastie / nfs rw 0 0

This, however, will only work if mount_nfs(8) is in your system. And (well, too bad) it hasn’t been packaged yet. So you can copy it from a FreeBSD base tarball, along with its dependencies (/lib/libc.so.7, /libexec and /etc/netconfig).

Step 6: Finally the system boots, but you’ll soon notice that file locks don’t work. This breaks some daemons (rsyslog), dpkg, apt and possibly a lot other things. Actual (i.e. non-local) file locking on NFS requires a running lock daemon, both on client and on server. However, since FreeBSD lockd isn’t packaged either, and you probably don’t need global locks anyway, I suggest you disable it with “nolockd” mount option.
The tricky part is that as explained in mount_nfs manpage, nolockd will only be honored when performing the initial mount, and it will be silently ignored if used while updating the mount options. So fstab is not an option.
The only way I could find of telling the kernel to use arbitrary options when mounting / is by sending them via DHCP. Use the following options in your dhcpd.conf:

option option-130 code 130 = text;
option option-130 “nolockd”;

Et voilà. A Debian GNU/kFreeBSD system running completely in diskless mode. Enjoy!

Finally, amd32 is taking shape

When AMD launched its now widely used 64-bit architecture in year 2000, it started marketing it as a significant step forward because of it having longer word and pointer size. One just has to check the advertising material and notice those big “64” being touted as the main improvement.

But it is commonly accepted that most applications don’t need a 64-bit address space for anything. Building them in LP64 model is just a waste of memory due to increase of pointer size. Even though this made most applications lag behind, the new architecture still was an improvement in terms of speed because of the AMD revised ISA, featuring changes like:

• The program counter register (%rip) can be accessed directly. This means that PIC (position-independent code) can be implemented sanely without doing strange, inefficient gimmicks (on i386, one had to perform a dummy “call” and retrieve the top of the stack inmediately afterwards).
• Add 8 new general-purpose registers to the first set of 8 registers, most of which were either claimed as implicit argument in some instructions or claimed by the ABI, and as such not really “general purpose”.

To summarize, most of the merit from AMD64 architecture was in fixing some of the insanity of Intel 386 ISA (instruction set architecture), rather than the increased pointer size which was a source of inefficiency most of the time.

Perhaps AMD didn’t evaluate this correctly, or perhaps its marketing side won over technical merit. However, although the new hardware is biased towards LP64 data model, it’s not actually enforced. It was a matter of time until an independent project took over and attempted to fix this, combining the AMD64 ISA with ILP32 data model.

I’ve been reading with much interest in the binutils and gcc mailing lists that such project is beginning to take shape. A port of binutils, GCC, GDB and Linux is already available. Future plans include porting Glibc which will make it possible to build a standard GNU derivative out of this.

Sadly, I don’t have the time to devote to this project myself, but I’ll continue following its progress. I’m looking forward to bringing this speed boost to my machines.

About ZFS in Squeeze (2)

Sometimes it’s nice being wrong. Contrary to what I predicted, ZFS will be supported in Debian Squeeze using the official installer.

This means that Debian Squeeze will be one of the first GNU distributions to support ZFS.

In fact, even though ZFS support didn’t make it to Debian-Installer beta1 by the time it was released, it is now available in the netboot images (this happens because netboot images fetch newer installer components from the internet).

As a consequence of this my unofficial installer can now be considered obsolete.

So why did I say something that turned out to be grossly inaccurate? It’s not due to anyone’s fault really. At that time, the version of Parted that included ZFS detection hadn’t migrated to Squeeze. The unblock policy didn’t appear to allow this migration. However, the Release Team kindly decided to make an exception that allowed this, and after Parted had migrated the changes in Debian-Installer itself went in quite smoothly.

Re: The Freaky Wall

Heya Sergio

Your blog doesn’t allow comments :-( so I reply here. First off, thanks for being one of the first Debian developers in trying out my modified installer with ZFS support, you guys made my day!

Some quick notes on the bugs you found:

The “missing //@ prefix” bug is #600578, fixed in upstream, in sid and in squeeze. Thanks :-)

The “boot partition stays in /target/boot in final install” bug is not in BTS. It was present in a preliminar version of partman-zfs which was never uploaded. The version currently in the archive correctly uses “altroot” option to fix this.

The “/lib/modules is needed for boot” problem was reported as #600568, but I haven’t been able to persuade the kFreeBSD maintainers that this is necessary. It’s not critical anyway.

Glad you finally got a working setup.

About ZFS in Squeeze

Update: About ZFS in Squeeze (2)

The bad news is that you won’t be able to use ZFS as your root filesystem in Debian Squeeze with the official installer. The blocker is missing support in GNU Parted. Unfortunately the patch I sent in August wasn’t integrated in time for the freeze (and still isn’t, but there’s no hurry now, it’ll hopefully be there for Wheezy).

The good news is that all fixes required for the installed system (including ZFS userland utilities and GRUB) made it and are already in Squeeze. As a result, the unofficial installer I built in September will (unless something weird happens) continue to work during Squeeze life cycle. Note that although the installer itself is modified, it installs a pristine Debian GNU/kFreeBSD system from official packages.

Hope this is helpful to you if you plan on deploying ZFS in your organization and the other approaches don’t fit well enough with your needs.