How many made the migration?

Our colleagues at WA research report: “As of today we have 122,187 users registered in 2,227 ‘active’ and 440 ‘archived’ Dgroups (however, some 17,707 of these do not seem to belong to any of the migrated groups, and we know that some email accounts are no longer active).

Every day we receive between 1,800 and 3,200 messages for dgroups.org, and send out between 50 and 90,000 to members.

So, despite a large cleanup of Dgroups late in 2008, the total graphs continue their generaly upward trend – this may change however after we further review and remove inactive email accounts.

As a heavy and long-time Dgroups user, Dgroups creator, and overall Dgroups coordinator, I have many interests in the new platform.

I was immediately very reassured by the reliable email system in the new ‘D2’ … in the past months, not being sure that messages would pass through ‘D1’was a continual worry, and a source of many many frustrated email messages from around the world. Good job D2!

One change I have noticed is that I seem to use the web much more to manage my D2 groups that I did on D1. Since email is at the heart of Dgroups, I hope this is not something permanent.

Logging on to D2 today, I was immediately struck by the many new arrivals on the new platform. The big migration of groups has begun. It feels a little bit like what it must have felt to be on Ellis Island in New York all those years ago! I look forward to meet up again with groups from the old platform, connecting and re-connecting.

One of my major concerns is to ensure that the new platform helps and encourages us migrant groups to meet up again, and not be lost forever in closed ‘communities.’ The current web interface certainly gives me a full screen of information, we need to ensure that basic information on all groups is open and that we do not encourage people to close themselves off. They can go to Google for that!

So far so good …

Peter Ballantyne

Alpha phase – at present we are making sure the Dgroups 2 platform allows one to perform all activities one is used to. Dgroups is all about email: consequently most of our work now is about tuning the new platform to offer what is needed to continue smooth operation mailing list operation.

This is purely software development and testing operation: we work against a list of requirements for the new platform making sure we cover all aspects of important functionality of the existing one. The work is divided in three functional areas: email, web user interface, and user profile features.

Beta test – as soon as we complete all the major features, we’ll open up the platform for an increasing number of existing Dgroups users to look around, try new things, see how their groups will look like.

At this stage, the platform will have all major functions available, but those will be rough around the edges. We’ll continue polishing and working as you look around. We’ll all communicate a lot about what you like or not and whether we’re missing something crucial that the majority of you were able to do with the old system. During this period, we’ll provide you with a copy of your data from the live Dgroups site, but only to look at – you will still use the existing live site to run your groups.

If you are feeling good about what you see and are tolerant towards technology, go ahead and create a few new groups using the new platform. Sure, it won’t be 100% finished, yet it won’t be crashing either.

Transition – when we’re sure it all works well, we’ll ask you not to create new groups on the existing live site, but to use the new one. Existing groups might still function on the old site for a while, but all new stuff goes through Dgroups 2. At this stage, we’ll have full support in place, all email messages will pass through the new platform and continue to the old one – thus the new one will be a mirror of whatever is going on with the current platform.

We want to give you some time to check the new platform and look around, learn the basics without pressure – your important groups will continue to run on the existing platform you already know so well.

Switch – once you get to know enough of the new platform to send and receive messages, approve new members, add resources, …, we’ll just flip the switch and immediately the new platform will start sending messages instead of the old one. All messages will already be there, we’ll copy all resources in advance. No downtime.

The Great Beyond – on February 20, 2009, we’ll shut down the old Dgroups system forever. That day on, we’ll work hard on new and exciting things to make Dgroups the best place on the Internet for international development community to conduct their dialogue.

So our priority has been to fully understand the reality of groups – how many are currently active, how many are dormant (not currently active but likely to be in the future) and how many should have been closed but weren’t? I have to confess here, for example, that when I was at OneWorld several groups were started for projects I was involved in that shouldn’t be still be there – but still are because I didn’t make sure they were closed. There is also the important issue of what we are calling ‘orphan groups’, groups that don’t currently connect to a current member of the dgroups partnership. Bellanet, for example, set up lots of groups. Those ‘owned’ by IDRC transferred out of dgroups to an IDRC server but lots of groups, including some large active ones, aren’t currently linked to a member.

We are clear that all groups that want to migrate to the new dgroups will have the opportunity to do so which is why we are spending a lot of time at the moment sorting out these issues. We have started the process by contacting the creators of dgroups, who are working through the lists at the moment.

We are also keen to re-activate the peer-support groups that were busy in the past but have been less so recently. Peer-support is an important way to share knowledge and spread limited resources between larger and smaller communities. We have contacted all the listed administrators: we are placing bets on how many bounces there are.

Meanwhile, we have also been focusing on the email system, making sure the new platform can support the email functions that have always been – and will continue to be – a central part of dgroups. The four of us in the migration team are experimenting with a trial environment at the moment and, as planned, we will invite a small number of people to join us over the next couple of weeks.

Pete Cranston

Our migration team is using this hierarchy to simplify communication. Our top-level group, ‘Migration to D2’ deals with general communication around migration. This group also serves as the entry point for all people we’re involving in the work around the migration process. Yet, we have some specialized topics, like data analysis, observations on the behavior of the new platforms, meeting coordination, and similar. For these specific topics that include only a subset of people involved, it makes sense to treat them separately. This is where the hierarchy comes in: we dedicated a sub-group for each of the specialized topics.

The interesting part is how the platform deals with hierarchy membership: all members of the sub-groups are always members of their parent groups, all the way to the top. If we invite someone to a sub-group, that someone is automatically a member of the parent groups (but not the sibling groups). Membership always propagates upwards. 

Conversely, an administrator of a group is automatically administrator of all sub-groups. Of course, one can assign a new administrator of a subgroup, who in turn can administer all sub-groups of that group, but is only a member of all parents. 🙂 Quite a mouthful to say, yet simplifies user management greatly.

There is no limit how many levels of sub-groups one wants to create, except maybe in practicality of writing an URL that is 1000+ characters long. Each group still gets its own mailing list and a document library, and outside of the URL and membership rules, it behaves like a top-level group.

This blog is the product of a meeting last week in The Hague. It brought together four members of the core project team – Damir Simunic (WA Research – the company selected to deliver dgroups 2.0), Hapee De Groot (Hivos), Mark Hammersley (leading on Communications, amongst other things) and Pete Cranston (Migration Project Manager), as well as Peter Ballantyne (Dgroups coordinator) and Christian Kreutz (Dgroups Executive Committee member from IICD).

After looking at the data from Damir’s initial analysis we outlined principles, phase plans and milestones for the next six months.

We agreed four basic principles:

• All groups, together with their users, will be offered the opportunity to migrate to the new system
• Migration will be organised around clusters of groups, organised by creators or members of the dgroups partnership associated with the group
• We aim to deliver a managed, exponentially increasing migration process
• We will communicate actions and progress, regularly, through as many channels as appropriate

Phases

Learning and Evolving: from now until the end of October, WA Research is building the first prototype application. Starting with the core team, we will be inviting some of the people who have expressed interest in testing to work with us in shaping that platform

Beta Testing: from 1st November for about 6 weeks, we will invite more people and their groups to participate in beta testing the platform. There will still be lots of opportunities in this period to help shape the final product.

Switch over: sometime in late December, the new platform will be complete and live – and some groups who have been part of the beta testing will operate completely in the new environment.

Migration: Late December into January 2009 – the majority of groups will migrate in this period

Completion and Decomissioning: during February, any remaining groups will be migrated to the new platform, and the orginal platform will be retired – to survive perhaps in the wayback machine.

If you have any queries or suggestions, please either write to me (pc AT euforic.org) or leave a comment here in this blog.