Core Tech Team

[Nicholas Kaye-Smith]

how many is a good number for a core technical team
with login permissions?  Three?  Six?  What?

Discuss.

[Nicholas Kaye-Smith]

Larry, by login permissions do you mean shell access, ftp access or mediawiki sysop status?

I suggest: not more than 3 people with shell/ftp access, and not more than 3 more people with sysop status. So about 6 people altogether.

[Larry Sanger]

Probably ssh access.

[Greg Sabino Mullane]

Probably ssh access.

Depends on who we have. Six for ssh sounds reasonable - you need at least a big enough pool to have good overlap of availability and areas of expertise.  More than that for sysops should be fine, although ideally there will be little for them to do that cannot be handled by the editors.

[Zachary Pruckowski]

Can I suggest trying to set it up so that you get someone with some sort of access on "the other side of the world"?  What I mean is that the Citizendium-l list worked really well because one of the moderators was in the US and one was in Australia, such that one of them was usually awake, with at most a few hours of non-overlap.  I suggest this because when we go to live time, we likely will want to have a situation where someone can react if something crashes/KPs/explodes, and it would stink to be down for 8 hours because the crash happened at 2 AM EDT, and all the techies are asleep, even though the problem is a relatively simple fix.

Then again, I'm have no experience at this, I'm just a 19-year-old CS student.

[Peter Hitchmough]

Just wait. 

Peter (GMT)

[Larry Sanger]

Well, I don't want to announce the group yet...soon...but three people have agreed.  I think two of them are in the U.S. and one of them is in the U.K. 

[Greg Sabino Mullane]

...
crashes/KPs/explodes, and it would stink to be down for 8 hours because the crash happened at 2 AM EDT, and all the techies are asleep, even though the problem is a relatively simple fix.

Who sleeps in till 9:50AM? Besides, I think a explosion would merit a few phone calls, regardless of the hour...

[Zachary Pruckowski]

...
crashes/KPs/explodes, and it would stink to be down for 8 hours because the crash happened at 2 AM EDT, and all the techies are asleep, even though the problem is a relatively simple fix.

Who sleeps in till 9:50AM? Besides, I think a explosion would merit a few phone calls, regardless of the hour...

OK, an explosion would merit phone calls.  many, many phone calls.  I meant that having an international tech-team gets you an awake techie at most hours instead of a hastily awoken one.

And for the record, when I get the chance (which is very rare) I sleep long past 10 AM.  But then again, that's because college and homework and everything affords me less than 6 hours of sleep 5 nights a week.

[Skapare]

How may people should have access at the core level? I'd say it should be as many as you can trust, but no more than you can manage. Certainly if you can trust 100 people that might seem great, but it is most certainly unmanageable. The stated figure of 3 seems a tad bit low to me, as that can risk perhaps too many unavailable in a crisis. I'd say 6. But this would be the steady-state figure once things get rolling. Before then might be more or less.

My understanding of core level are the people that would need root access to do what they do. You have to either trust the system (to not let untrusted people do bad things) or you have to trust the people (to not do bad things in a system with no trust structure imposed on them ... such as root would have).

I've run servers with multiple people having root access before. You have to understand that at this level, someone intent on doing bad things will, and will be able to hide it if they know what they are doing. But that shouldn't stop some specific practices to help manage the (trusted) root staff, and track activities ... not because anyone is untrusted ... but because others simply need to know what is going on.

Access to root should obviously be very limited. There are also some things to do to manage this access better:

1. Do not run SSHD on port 22. Port 22 is the target of tons of random hacking attempts. It will flood your log even from attempts that don't know the nature of the project they are hitting. Add to that the high profile Citizendium will soon have, and you can expect even more attempts. Flooded logs make it harder to track real problems. So pick random ports for SSHD and communicate that only to the appropriate people.

2. Keep OpenSSH and OpenSSL up to date first before just about anything else.

3. Compile OpenSSH statically. Make separate copies of SSHD for #4.

4. Run 2 or 3 separate instances of SSHD on separate randomly decided ports (never port 22 or 23). The redundant instances will allow some flexibility in taking down a problem instance. Run them each from their own distinct copy of SSHD statically compiled. This will provent certain problems of updating OpenSSL from locking everyone out of the machine.

5. Never login directly as root. Always login as your own username, and then "su" to root.

6. User different passwords for root for each user. This is done by having extra root names per user, like "root-" followed by 3 initials. They would all have uid 0, but different passwords. The advantage of this is if you do have to remove someone, you don't have to do a panic password redistribution of a single root password (doing things in panic mode is bad). In such cases, people should change their passwords, but there is a little breathing room for this, this way.

Note that I am not saying these things should be all done immediately. Of course sooner is better. But I'm thinking more along the lines of when things start to expand beyond the initial launch.

[Greg Sabino Mullane]

4. Run 2 or 3 separate instances of SSHD on separate randomly decided ports (never port 22 or 23). The redundant instances will allow some flexibility in taking down a problem instance. Run them each from their own distinct copy of SSHD statically compiled. This will provent certain problems of updating OpenSSL from locking everyone out of the machine.

5. Never login directly as root. Always login as your own username, and then "su" to root.

6. User different passwords for root for each user. This is done by having extra root names per user, like "root-" followed by 3 initials. They would all have uid 0, but different passwords. The advantage of this is if you do have to remove someone, you don't have to do a panic password redistribution of a single root password (doing things in panic mode is bad). In such cases, people should change their passwords, but there is a little breathing room for this, this way.

#4: Not necessary, except when you are actually doing the update.

#5: Don't use su, use ssh keys. More secure, more convienent.

#6. Using ssh keys solves this problem too. Removing someone becomes as simple as removing a line from the authorized_keys file.

I don't think running ssh on a non-standard port really buys you that much, and can cause trouble with firewalls, but I'm not totally opposed to it either.

[Skapare]

See how easy it is to change the times in the quotes 

#4: Not necessary, except when you are actually doing the update.

But you need to have it already configured and ready to go. It's no time to test a new configuration of a critical element like being able to access the machine when also under the stress of trying to plug a security hole. Then it's really no issue to just leave them all running.

#5: Don't use su, use ssh keys. More secure, more convienent.

Do both. Unless you want to patch SSHD every version to do better logging.

#6. Using ssh keys solves this problem too. Removing someone becomes as simple as removing a line from the authorized_keys file.

If you can make SSHD good enough to avoid the su. OOB, SSHD really isn't in terms of administrative utility. Some hacks would make it good enough, but that's not an area you want to go to. I used to run hacked SSHD, and I know the hassles.

I don't think running ssh on a non-standard port really buys you that much, and can cause trouble with firewalls, but I'm not totally opposed to it either.

Having been heavily flooded by kiddie dictionary attacks on SSHD before, I can say there sure is value in eliminating at least the trivial attempts. A high profile project will no doubt get attemps no matter what the port. But at least you'll know you are specifically targeted on those, as opposed to the hundreds of "drive by shootings at port 22" that occur daily.

I've compared the level of attempt with port 22 on and off. It does slow down with port 22 off, so it seems the attempts are stateful. They don't come back as often if the port is dead. Another possible option is a knock-knock protocol.

[Jason "Electrawn" Potkanski]

#4: Not necessary, except when you are actually doing the update.

But you need to have it already configured and ready to go. It's no time to test a new configuration of a critical element like being able to access the machine when also under the stress of trying to plug a security hole. Then it's really no issue to just leave them all running.

#5: Don't use su, use ssh keys. More secure, more convienent.

Do both. Unless you want to patch SSHD every version to do better logging.

#6. Using ssh keys solves this problem too. Removing someone becomes as simple as removing a line from the authorized_keys file.

If you can make SSHD good enough to avoid the su. OOB, SSHD really isn't in terms of administrative utility. Some hacks would make it good enough, but that's not an area you want to go to. I used to run hacked SSHD, and I know the hassles.

I don't think running ssh on a non-standard port really buys you that much, and can cause trouble with firewalls, but I'm not totally opposed to it either.

Having been heavily flooded by kiddie dictionary attacks on SSHD before, I can say there sure is value in eliminating at least the trivial attempts. A high profile project will no doubt get attemps no matter what the port. But at least you'll know you are specifically targeted on those, as opposed to the hundreds of "drive by shootings at port 22" that occur daily.

I've compared the level of attempt with port 22 on and off. It does slow down with port 22 off, so it seems the attempts are stateful. They don't come back as often if the port is dead. Another possible option is a knock-knock protocol.

Isn't the easy solution to that to just require keys and ditch SSH password authorization? (I see turnstep already said this three times).

Links to a best practices or howto document would be a good idea?  I am not up to date on linux server hardening tactics.

-jtp