I agree a lot with what russ said, just the real money=game money P2W should be gone. Heres my suggestions:
I say limit the max players in a faction or make it the more in a faction, the more you have to pay or something similar, as we don't need factions including half the server population.
Make it so you have to pay for each piece of land you claim making raiding someone require you risking your money, for possibly no return. Also making it more challenging to claim as much as your power allows, I believe factions has an exponentially growing pay per land built in.
I also would like to see a rule making bases having to be either above ground, or having an entrance that can be used if claimed, no base 100 blocks under an ocean that would require digging and someone using xray to find it.
Due to changes in nether in 1.5 we would need to make a new world, is there anyway possible to make it that the closer you are around spawn the more ores there are, and if your far diamond could be hard to find? That would keep people from going to the world max size and hiding there, and would force people to be closer in and lead to some nice battles.
Another one like russ said, no admin shops, I don't even really like having a central trade hub. As a replacement we could have chest/sign shop access for everyone, but in order to use it you would be required to build a shop for it that looks nice, not just plopping a chest down and saying "Buy here".
A world limit that's a bit bigger, or including mushroom biomes inside the world limit, Mycelium allows for easy mushroom farming, and mooshrooms can be a great food source if you can manage getting them to your base.
That brings me to my next point, Mobcatcher. Its cool, but should cost money if possible or require a diamond per mob caught to add some challenge.
No auto repairing TNT/creeper holes. I'm not saying allow someone to blow up an entire base, I am saying allow them to creeper bomb/TNT cannon into a base and raid it. TNT is hard to acquire in large numbers, and with it auto repairing its pointless to make. This server should be challenging, so lets give players one more thing to fear.
Allow flowing lava, tons of cool things are possible with it, such as auto repairing walls and cobblestone generators. Minecraft is a sandbox game, allow people to mess around with the tools they are given. I doubt its gonna be used to grief, and lava can be disabled for members/guests should it be an issue. Firespread however should definitely stay off :P
Definitely make money harder to acquire, you shouldn't be able to start a faction within the first hour of play, and thus adding a good starting challenge before you can make a faction to protect your items. Maybe you can add PvP protection to new players for their first hour, that you can opt out of, to make it so they don't have to worry about players until they should (hopefully) afford a faction.
Add the beheading mod, allowing player heads to sometimes drop upon a kill by a sword. Trophy room anyone?
The servers fun, but it offers no challenge. Hopefully these ideas can make it a challenging world and would require teamwork to make it to the top. This is all I can think of right now, If I think of anything else I will add it on. Feel free to voice your opinion/add on to what I said so Nishok and Silent can make this server fun for much longer time!
Mods I found that can accomplish some of this:
StopLava - Bukkit This may still work, would eliminate fear of lava griefing from guest/members by only giving lava buckets to sup+.
HeadBounty - Bukkit Currently down for bugfixing, but would allow you to put a bounty on someones head.