This is the jabber.org service notice for Friday, January 28, 2010.

For the most part, today was a good day at the jabber.org IM service. We installed a new build from Isode earlier today that seems to have solved the login delays our users experienced, as well as most of the memory usage issues. This build isn’t perfect, because we had a hard crash this evening, followed by some false restarts, so we’ll be monitoring it closely over the weekend. In addition to these fixes, the jabber.org admin team corrected a problem with our digital certificate and continues to debug connection issues with a number of different IM clients.

We will post updates over the weekend, so stay tuned.

This is the jabber.org service notice for Wednesday, January 27, 2010.

There’s not as much to report today. The Isode team continues to track down the memory issues as well as a few less significant bugs and feature requests (DIGEST-MD5 login issues, vCard fixes, enabling chatroom logs, etc.). On the jabber.org team we are working to diagnose connection problems experienced by some users of Finch, Gajim, GNU Freetalk, Kopete, Meebo, Palringo, and a few other clients (most of the issues seem to be related to our disabling of the old SSL-only port 5223, but there might be other bugs involved). We also discovered an IP address that was making about 50 TCP connections a second without ever trying to start an XMPP stream, so we have blocked that IP address. We’re on the lookout for other abusive traffic, which might be causing more general problems at the jabber.org service.

Today we also held an impromptu meeting of jabber.org users in the jabber@conference.jabber.org chatroom. If our users would like us to hold a weekly meeting for a while, drop by the jabber@ chatroom and let us know (you can even join via the web as described on the help page).

Tomorrow (Thursday) we plan to fix a problem with our certificate chain, deploy several fixes from the Isode team, and track down more of the client connection problems mentioned above. We’ll keep you posted via identi.ca (mirrored to Twitter).

Peter Saint-Andre

We have reports that older versions of Kopete have trouble connecting to jabber.org after the server migration. These issues do not affect Kopete on KDE 4, only on KDE 3. Kopete 3.x does not support STARTTLS (encryption upgrade on port 5222) and right now jabber.org does not support the old SSL-only port 5223, so you will need to disable SSL connections via port 5223 and connect unencrypted on port 5222. Unfortunately that is *not* secure and we do *not* recommend unencrypted connections (in fact we might forbid unencrypted connections in the next few months). However, we also have reports that even unencrypted connections are not working for all KDE 3.x users. We are working to debug the problem and will post again once we have determined the cause. In the meantime you might consider upgrading to KDE 4 (yes, we know that is a big upgrade) or using another Linux or cross-platform client, such as Gajim or Psi. We apologize for the inconvenience.

UPDATE: It seems that there are issues with KDE 4, too. We are working to isolate the problem and will report back as soon as possible.

UPDATE #2: A fix is in place for a server-side bug here, and we have reports of successful connections using versions as early as Kopete 0.12.

This is the jabber.org service notice for Monday, January 25, 2010.

We made good progress today in solving a number of open issues. The Isode team found and fixed several critical bugs today and over the weekend. The jabber.org admin team cleaned up contact lists for the ~250 users whose data was corrupted last Saturday, instituted several automated monitoring, reporting, and data backup tasks as well as log rotation to help prevent further problems, and has committed to improved communication, including notices throughout the day via the identi.ca microblogging service.

Some of these code and configuration fixes have required restarts of the service. Our users should expect that to continue for several more days, but we will attempt to announce all restarts beforehand on identi.ca.

We still have several open issues to address this week, and we shall post about those in the coming days.

Thanks again to our users for their patience.

Peter Saint-Andre

First, my apologies. The jabber.org IM server is run by a small team of volunteers, but that is no excuse for the poor level of service you have experienced lately.

Our failures have been many:

1. We thought that migrating to new server software would alleviate some of the problems we had in the past. Clearly the new software we’ve been trying has been even less reliable so far, resulting in frequent outages, random disconnections, the inability to log in, and other problems. I think that most or all of the problems are transitional and related to the difficulty of migrating a large service to new software, but the fact remains that the user experience has been horrible over the last week.

2. We have made our share of mistakes (unrelated to the software we use) regarding data backups, software deployment, service monitoring, and other best practices for a large public IM server.

3. We have failed to communicate clearly and regularly regarding the status of the server.

These failures are unacceptable for our users and embarrassing for the jabber.org team. We need to do better. In particular, we commit to:

1. Deploying software that works. We are collaborating with the developers of the software we’re currently using (the M-Link product from Isode Ltd.) regarding some bugs in their code. Although we don’t blame them for the bugs (all software has bugs, and the jabber.org service receives a lot of strange and unpredictable traffic from XMPP clients and servers that connect to our service), the bugs need to be fixed and the crashes need to stop. Much as we like and respect our colleagues at Isode, we have also looked into alternative software solutions because our first commitment must be to our users. I think we’ll be able to successfully complete the migration to M-Link, but at the same time it would be irresponsible of us to not consider other possible options in case they become necessary.

2. We are instituting improvements in our processes and procedures so that we can prevent problems from arising in the future, and so that we can respond more quickly when problems do occur.

3. We shall post a daily status update at www.jabber.org for the foreseeable future to keep our users informed.

We have been offering free IM accounts at jabber.org since 1999, and we want to provide a service we can be proud of. Lately there has been very little pride on the jabber.org team.

Because I am ultimately responsible for the jabber.org server, I am personally embarrassed by the poor service we have been providing, and I shall do whatever it takes to ensure that you, our users, will receive much better service in the future.

Peter Saint-Andre