Thursday, September 1, 2011

Why is the packaging process so broken?!?

For those that do not know, I develop software for the Mythbuntu project. Over the past couple of Ubuntu releases I have been working on Mythbuntu-bare, a backup and restore utility for MythTV systems. I have been slowly adding functionality to this software which was included with the mythbuntu-common package, but due to the growing complexity of the software and size I decided it was time to move it from the mythbuntu-common package into its own set of packages. This is where my issues began.

I originally uploaded the new package to REVU (the process for getting packages into the repository) on July 19th, 2011. Feature freeze was on August 11th, 2011 so this should have been plenty of time to get the package reviewed. At the time I knew of one member of the Mythbuntu team that could review the software, and I worked with him to make sure everything was packaged correctly. Once everything was fixed I started asking for a second review in the #ubuntu-motu IRC channel, as each package needs to pass two reviews in order to be uploaded to the archive. At this point I had about 1-2 weeks left until feature freeze. Every few days I would ask in the channel without getting much of any assistance, although I was told I needed to file a needs packaging bug, which I promptly did and subscribed the requested team. However this did not seem to help much. At the beginning of August another member of the Mythbuntu team stepped forward and was able to give me the second review I needed, after a few additional changes. On August 11th (yes the day of feature freeze), my software was uploaded to what I believed to be the archive.

On Friday, August 19th, I received an email from a Canonical employee stating that my package was being rejected (with a list of reasons) and that I should feel free to adjust the package and reupload. The reasons listed in the email are either very minor issues (optionally need fixed) or do not apply to my package. I happened to be at the end of a week long vacation at this time, but fired off a reply 12 hours after receiving the email and explained why each one of the list items did not apply. Although just yesterday I tracked down the Canonical employee on IRC and they said I would get a reply that day, I still have not received that a reply.

At this point I wonder what else I could have done. I submitted my package to the REVU process with plenty of time before feature freeze. I pleaded for it to be reviewed by a MOTU in the #ubuntu-motu channel, but had to bug people I know to get it reviewed in time before feature freeze. Then even after following these rules and fixing my packaging when told to do so, it still gets rejected by an archive admin. Do that admins not trust the MOTUs to review software? If so, why is the REVU process in place? I am starting to understand why there is a lot of software that is not packaged, as the REVU process seems to just be a way to discourage people from packaging things. I will be the first to admit that I am not the best programmer, but this process seems asinine. No matter how many hoops I seem to jump though there is always one more, and nobody will answer my questions about these hoops. As Ubuntu 11.10 Beta was just released, I believe the window to get this software into 11.10 has passed.

Maybe I am wrong and I just don't understand the big picture behind why Ubuntu needs all these levels to get though just to get software into the archive. After this process, I just don't care anymore. I just want to write software.


  1. Well, I'm no Ubuntu project strategist—and don't take this as sculpted in marble, please—but from talking with lifeless/mpt/mvo, it seems that the nearish future holds some changes for Ubuntu packaging, hopefully in a good direction.

    The essential vision is that third-party devs interested in providing software for Ubuntu create PPAs, which release updates at their discretion. PPAs would be indexed in the Software Center once approved. In your case, mythbuntu-bare would probably go in a Mythbuntu-run archive.

    There are other visions, of course, but a number of prominent voices in Ubuntu agree that the current process is overly byzantine. Things should get better, but there's little use gussying up a broken system while a new shiny one is readying to replace it.

    For reference...

  2. That was what was originally thought when it was announced, but all the research I've done on it indicates it's only for people selling software in the software center. For people like me, it will remain broke. Note: I'm currently at work, so I haven't seen the linked youtube video yet.

  3. j-johan-edwards, I don't think it will be just as easy as that, unless there is a major shift on Ubuntu's central repository management philosophy which was not announced yet. One of the key advantages of using a central repository versus a distributed managed-by-many distribution is that you can actually enforce a QA process which is expected to warrant packaging quality and reliability. If they really chose that venue for the general open source repository, oh boy, I am sure many people will feel to have wasted a major effort meeting quality «some time frustrating» requirements for such a long time.

  4. Right, let's deal with this section by section:

    Yes, I agree, REVU isn't working for getting new packages in. That's not REVU's fault, though. There aren't very many active MOTUs, and we are all rather overworked as it is. We simply don't have the manpower to maintain universe and review new packages.
    In fact, part of the problem in MOTU is that there are quite a few packages in Ubuntu Universe that aren't in Debian because they came in through REVU. If the person who initially added it has lost interest (or is busy) the maintenance of these packages falls on MOTU's shoulders. So, as you can imagine, Ubuntu tries to discourage people from using REVU, and rather bring new packages in through Debian.
    Of course, this package sounds Ubuntu-specific, and therefore a good candidate for REVU, but we don't have many people actively reviewing on REVU. I saw your requests for review on IRC, sorry I didn't have a look. Fortunately, you could get other Mythbuntu people to review it.

    NEW Queue:
    You said you thought your software was uploaded to "what I believed to be the archive". It was the archive, but all new packages have to go through the "NEW" queue, and be checked by Archive Admins, before they land in the archive itself. The point of this is to pick up big, serious problems (such as licensing).
    Another reason for preferring new packages through Debian, is that the Ubuntu Archive Admins can trust the Debian ftpmaster team to have done a thorough NEW review in Debian (this makes NEW review in Ubuntu a lot quicker).
    The Canonical Employee you received the mail will have been wearing an Archive Admin hat, not a Canonical hat. (Most Archive Admins work for Canonical, as most Ubuntu Developers do, these days :/ ) They tend to be pretty busy, especially during feature freeze, which is why we like to get new packages in earlier in the release cycle.

    The Package itself:
    I don't know what the reasons for the rejection were, but I just had a quick look at the package. Lintian shows up quite a few warnings that are probably worth dealing with. More importantly, it doesn't look like you are doing anything to get the .py files byte-compiled (I'm happy to help with that).
    This is also a daemon, running as root, so it's understandable that there'd be extra scrutiny. (I haven't looked at any of the source code, just the packaging)

    The big picture:
    This bureaucracy is because Ubuntu needs to ensure it maintains a high quality archive. Unfortunately we (MOTU) don't have enough manpower to manage that with the packages we already have, never mind new ones, so we could do a better job of that. I know you just want to write software (don't we all), but we also don't want to release packages that are going to be problematic.

    Sorry for the long reply :/