Bam-zine

Networking changes on TQ

Well, we have done tons on upgrading PC machines with more memory faster hard drives, gone from floppy drivers to some RAMSAN thing, but while this was happening the network operations team has been looking at the most important thing (from the NOC standpoint), the network. Without a network connection to this interweb thing, EVE would be a great single player console game for some lonely guys in Telecity datacenter ;P

The Internet, how does it really work (yikes!!)

When I started working with Internet routing, coming from a rigid corporate environment, I was surprised how little control you have over your traffic and how flaky the Internet really is. I have read in books about all the bells and whistles of traffic engineering but I’ve never seen the real application until I met BGP and the Internet.  As many of you know, the Internet started out as few universities talking to each other. BGP (Border Gateway protocol) is the main building block of the Internet. The Internet is built up by BGP speaking peers (routers). Each origin (like CCP) has a network slice and we announce our network to our neighbors and hope they announce them to other neighbors.  We rely on our next hop (upstream) neighbors to accept our prefix and carry the traffic.

The major problem with the Internet is that BGP is not very deterministic; it relies heavily on something called an AS-PATH. Each entity on the Internet has an AS number (autonomous system). The path from client to CCP is usually the AS of client ISP, Tier1 SP (top of the Internet), CCP. There will often be other upstream providers of the client ISP. What is the problem here? Within one AS there could be 10 router hops (as seen via traceroute) Those hops could go to Europe, the US and then back to Europe. The best part is that there is a lot of meddling with the best paths. Some ISP’s will have a good connection to some Tier1 providers, but may also have a cheaper connection with some lousy provider that they engineer the traffic to.

One example of this problem is Vodafone in Iceland. They peer with a large transit network called Cogent and then with T-systems (Deutche Telecom) in Europe. We don’t peer with any of them. The shortest path from CCP to Vodafone Iceland is CCP-Telia-Cogent-Vodafone Iceland. The AS path, then, is two hops. The problem with this is that Cogent chooses to send all traffic to US and then over to Iceland via Vodafone. This is one AS hop.  What we must then do is to manually engineer the traffic to use a longer path through our multi-homed connection in Telecity to get to Tiscali, who peer with T-systems, and then we reach Vodafone using the shortest real distance. 

So where does CCP stand

Until recently, we had two connections to the Internet: one was with Telia Sonera, a Tier1 service provider (http://en.wikipedia.org/wiki/Tier_1_network). The other connection was with a multi-home peering platform operated by Telecity in London. We had limited control over our traffic. One important thing about BGP: you can control where you send the traffic, not how it returns.

So remember when you do a traceroute to TQ, it’s very likely the traceroute from TQ could be very different. (I’ll discuss that more below).

We decided it was not enough having only two providers even if they are dual connected for redundancy. Internet services providers get overloaded and have all kinds of issues. Having only two limits the options we have circumventing bad spots on the Internet. It was decided to get as close to the user as possible. This meant peering with the Tier1 service providers directly and in some cases try to obtain direct peering with big end user service providers. After doing some searching we found a company called Packet Exchange. They are supplying us with direct peering with some of the major Tier1 providers and direct peering with 120 smaller SP’s. This was a very good match for us and we jumped on it.

The hardware conundrum

TQ core network infrastructure is built up by Cisco Catalyst switches and routers. Up to now we have been using a Cisco Catalyst 6500 as our distribution switch when peering with the outside world. When this hardware was chosen, the Internet routing table was around 250k entries and this equipment could handle it with ease. The table is growing day by day and is close to 280k routes now. The router module in the 6500 is not specialized to handle large number of BGP routes, so it became apparent that if we’d want to increase the number of peers we would have to change the network structure.

Initially we looked at few different platforms. It was always clear that we would stick with Cisco as we have good internal knowledge of the platform and they have proven to be very reliable. In the Cisco product line there are few models that we could use, one being the old and proven 7200 series, the newcomer ASR1000 series running IOS XE software and the supersized 7600-S chassis.

As we are CCP, we went for the supersized option and got the 7600-S platform with a RSP720-3CXL route processor. I read on some thread that it was indicated an IOS-based platform would give us challenges. I’m a true believer that IOS-based platform is perfect for us. The setup is not that complex, and now we have a very robust routing platform with a very fast switchover in case of hardware failure. Cisco 12K, CRS-1’s are in the big service provider range and are just so expensive and for our scenario offer no real additional value.

Last week:

We installed the 7600’s and turned on the links to Packet Exhange. This was done softly so we eased the changes in and hopefully no one noticed anything. There was a small hiccup as one of the fiber patch cables failed horribly but that was quickly dealt with.

We now have direct peering with: Global Crossing, Teleglobe, KPN, Telia, AboveNet and Level3. We still have the multihomed connection in Telecity operational, but that one is going to be closed soon. Level3 is eating up most of the traffic as expected as they are by far the largest Tier1 service provider.

We will soon turn on peering with 120 smaller SP’s through a multilateral peering platform that PacketExchange operates.

We are in fact a small Tier2 service provider, with one customer, TQ ;P (though we don’t have any downstream providers).

Next steps:

We continue working on improving the network infrastructure to get as close to you as possible. We are now looking at providing a “looking glass” service. This will allow you to take a look how the traffic is going from TQ to you. This is very important tool to find out if the traffic is perhaps taking some long detour as the traces you are currently supplying in the disconnect threads don’t telling the whole story. We now have a real option of rerouting traffic and choosing the best path to the customer as with only two providers the options where very limited. Perhaps one day you can choose the outgoing provider through the web portal ;P  Now, that would be very cool!  But, these are still under investigation on how, if, and when we should do this.

Just few notes about traceroute:

When you do a traceroute, you send a packet to the destination with a low time to live number in the header. A router in the path will subtract one from the TTL and if the TTL Is 0, it drops the packet and sends an ICMP message back saying “time exceeded.” Traceroute starts sending a packet with TTL of 1 and increments each time to discover the hops. When we see packets dropped in a traceroute, high latency numbers etc., it does not always indicate a real problem. Most Internet border routers are “big” routers; they will have something called the forwarding plane, which is the part responsible of sending/receiving the packets passing through the router. Then we have the control plane, which is the “router.” Even in a 7600 it’s only a 1200Mhz RISC processor, and it’s the one that is responding to pings and sending TTL expired messages. As with anything, TTL expired has been exploited for a DoS attack on routers so it’s usually rate limited; so is ICMP and other stuff that could impact the control plane of the router.  The router will answer when it feels like answering these messages. If you ping a router you might see jumps in response time and that is by no mean indication of how it is performing. It could be some indication on the housekeeping of the control plane, but not if it’s overloaded. “Routers are usually lazy answering pings” ;P

I hope this first blog from the network team helps explain what we are doing, the problems we’re facing and how we are tackling this ever changing thing called the Internet.

Art – Tech3 and the Renovated Effects

Art – Tech3 and the Renovated Effects

The Tech3 art team recently wrapped up the last stretch of the fourth variation for the newly discovered Strategic Cruisers. It’s been a crazy few months for the team and we’ve seen people grow from mere juniors into invaluable team members, as well as getting a helping hand from seasoned veterans like Art Director Kári Gunnarsson and Lead Illustrator Börkur Eiríksson. We’ve overcome many obstacles, fighting the technical restrictions, while also demanding quality and variety from the designs and their execution. Boundaries were pushed, polygons and pixels abused to the max. With time and resources having been at a premium, we truly had pushed it to the limit.

For those interested in the process, the video below depicts it to some degree, from sketches to blockout models to what we call paintovers to finished assets. Making the thousands of man hours put into these seem like mere minutes.

For those wanting to know more about the stats and game design behind the fourth variation, there’s a great discussion thread on the subject on the EVE-O forums.

The process had also been an evolving one. We had set out to make the asset production as efficient as possible but we also made sure that we were never sacrificing quality or vision to do so. Great care had been taken to insure all the varying pieces fit together. That’s no small task, seeing as we had 243 different ship combinations per race (972 total) with the with the 3 variations available when Apocrypha launched. The 4th variation made available as of Apocrypha 1.1 (April 16th) brings that total up to 4096 different ship combinations. That being said, we can’t exclude the possibility that some might find some interesting combinations not specifically to their liking, be it stat wise or visually.

By this time in our minds here in the Art department, we were certain that now close to 6 weeks after release the market would be flooded with Tech3 ships and people would be bored with the 3 variations already in and would be anxiously awaiting the 4th. This also describes the mentality during such a development phase of wanting to finish all the assets before launch. But it seems the required ventures into Wormhole space, with the accompanying Sleeper encounters as well as the manufacturing involved, has proven to be quite the challenge even for the most seasoned of players.

The experience has been quite the learning process and even though proving painful at times it ultimately yielded the results we were after. Not enough can be said about the efforts and dedication of the art teams involved and it’s been an honor to have been part of such a team that took a near impossible task and made it happen.

Effects

The effects system was due for an overhaul and we at CCP took that to heart. All the effects were renovated making eve look even greater than ever before. Having said that, the cliché is also true, there is never enough time to polish and iterate on the work. Add on top of that the fact the effects system itself also received an overhaul this time around. These facts left us with a handful of renovated effects after release, needing exactly that. More time and polish. With resources still being scarce we’re working from a priority list. On it were f.e. the Warp effect, POS forcefields and the SalvageScan effect. The stargate jump effect was also polished a bit, but mostly timing wise along with some technical defects being solved. Adding new effects alongside polishing our current ones will be an ongoing and evolving process as our artists get a better handle on the new system and figure out new and creative ways to abuse it. All in the noble goal to make EvE the most beautiful (SciFi) MMO out there.

“The Edge… there is no honest way to explain it because the only people who really know where it is are the ones who have gone over.” – Hunter S. Thompson

Fly safe,

- CCP jBot

CSM3 Candidates Application Breakdown

It is getting closer to election to the third CSM in EVE and the application period has now been concluded.

Following are the results from the candidacy application period.

The statistics.

• We received 56 applications.
• We accepted 40.
• We rejected 16.

Like with CSM1 and CSM2 candidacy applications, the main reasons for rejections were passport issues (either not supplied or expired), EULA violations (mostly ISK buying), one test application to ensure the functionality of the system and then Internet lolsters supplied a few applications.

• 3 females and 37 males are running, making the female applicants 7.5% of the total pool. This is an increase from the last election and higher than the overall player ratio, where the female population is 4.41% of the whole (according to user account information).
• The age range of the candidates is between 21 and 53 with the average age of 31 years.

The list of applicants will be published Monday the 27^th of April and will be viewable from this section.

We look forward to the campaign and we wish the best of luck to the candidates in the upcoming elections which will be open between the 5th of May and 19th of May 2009.

Trouble explaining the "EVEness" of EVE?

Got some friends who are curious about EVE but you just can’t wrap your head around how to describe it to them? Know someone just about to get into the game who could use a helping hand?

Destructoid recently featured a great article that explains the unforgiving nature of EVE and what sets it apart from other games. It manages expectations well while highlighting some of the core design principles. And not a bad read for veteran players as well. All in an easily forwarded link.

Trouble explaining the "EVEness" of EVE?

Got some friends who are curious about EVE but you just can’t wrap your head around how to describe it to them? Know someone just about to get into the game who could use a helping hand?

Destructoid recently featured a great article that explains the unforgiving nature of EVE and what sets it apart from other games. It manages expectations well while highlighting some of the core design principles. And not a bad read for veteran players as well. All in an easily forwarded link.

Tech3 and the Updated Effects

Tech3 and the Updated Effects

The Tech3 art team recently wrapped up the last stretch of the fourth variation for the newly discovered Strategic Cruisers. It’s been a crazy few months for the team and we’ve seen people grow from mere juniors into invaluable team members, as well as getting a helping hand from seasoned veterans like Art Director Kári Gunnarsson and Lead Illustrator Börkur Eiríksson. We’ve overcome many obstacles, fighting the technical restrictions, while also demanding quality and variety from the designs and their execution. Boundaries were pushed, polygons and pixels abused to the max. With time and resources having been at a premium, we truly had pushed it to the limit.

For those interested in the process, the video below depicts it to some degree, from sketches to blockout models to what we call paintovers to finished assets. Making the thousands of man hours put into these seem like mere minutes.

For those wanting to know more about the stats and game design behind the fourth variation, there’s a great discussion thread on the subject on the EVE-O forums.

The process had also been an evolving one. We had set out to make the asset production as efficient as possible but we also made sure that we were never sacrificing quality or vision to do so. Great care had been taken to insure all the varying pieces fit together. That’s no small task, seeing as we had 243 different ship combinations per race (972 total) with the 3 variations that were currently in game. The 4th variation brings that total up to 4096 different ship combinations. That being said, we can’t exclude the possibility that some might find some interesting combinations not specifically to their liking, be it stat wise or visually.

By this time in our minds, we were certain that now close to 3 weeks after release the market would be flooded with Tech3 ships and people would be bored with the 3 variations already in and would be anxiously awaiting the 4th. It seems the required ventures into Wormhole space, with the accompanying Sleeper encounters as well as the manufacturing involved, has proven to be quite the challenge even for the most seasoned of players.

The experience has been quite the learning process and even though proving painful at times it ultimately yielded the results we were after. Not enough can be said about the efforts and dedication of the team involved and it’s been an honor to have been part of such a team that took a near impossible task and made it happen.

Effects

The effects system was due for an overhaul and we at CCP took that to heart. All the effects were renovated making eve look even greater than ever before. Having said that, the cliché is also true, there is never enough time to polish and iterate on the work. Add on top of that the fact the effects system itself also received an overhaul this time around. These facts left us with a handful of renovated effects after release, needing exactly that. More time and polish. With resources still being scarce we’re working from a priority list. On it were f.e. the Warp effect, POS forcefields and the SalvageScan effect. The stargate jump effect was also polished a bit, but mostly timing wise along with some technical defects being solved. Adding new effects alongside polishing our current ones will be an ongoing and evolving process as our artists get a better handle on the new system and figure out new and creative ways to abuse it. All in the noble goal to make EvE the most beautiful (SciFi) MMO out there.

“The Edge… there is no honest way to explain it because the only people who really know where it is are the ones who have gone over.” – Hunter S. Thompson

Fly safe,

- CCP jBot

Tulips

Tulips, originally uploaded by Bambi_72.

Spring is definitely here now. Sunshine, flowers, insects…..

Raining

IMG_1596, originally uploaded by Bambi_72.

I did say there would be more pictures…

Water drops on the washing line.

I’m still here

What month is it? Doesn’t time fly when you are busy, and man, how busy have I been recently? I got hooked on this historical Earth Simulation called R.L, so far into it we have a baby due soon, hence my absence really.
So, project Anshar is still under way, its slow going when you are doing these big builds solo, especially with the quantities of minerals and advanced materials that needed sourcing. I managed to blag free blueprint copies from an old buddy who shall remain nameless (Cheers Ore!), this helped keep the build cost to a minimum, along with a well researched blueprint for the Obelisk. I currently have all my factory slots churning out the Capital Components for the Obelisk while associates of mine source me the vast quantities of Advanced Materials I need to build the Advanced Capital Construction Components. Its a shame the illegal Fermionics cartel got busted a while back and pushed the prices thru the roof, it does look like they are starting to fall again slowly. Still, there is a nice fat profit margin to be had on an Anshar.

I have been intrigued by the recent discovery of these ‘wormholes’ that have suddenly appeared everywhere, new frontiers always mean new opportunities to make money, so I hooked up with a few old buddies and we went on a few trips thru these wormholes into unknown space. It is a very strange place, and the locals are most definitely hostile. They have ships and weapons that I have never seen before, much more advanced than the best in our so called Empire. They do drop interesting loot when you finally crack them, including new technologies, which I am sure I could teach myself how to replicate and build. These parts are fetching a premium price on the open market for the time being, so probably better to sell them now and work with them a little later.
Anyway I must get back to work, I have some stealth bombers to put together….

New Rock, Paper, Shotgun Contest

Want your very own copy of the EVE Online retail box? Rock, Paper, Shotgun is giving away six copies featuring a 60 day game time card, an EVE career planner, your very own Interbus Shuttle and more. “How can I win such a fabulous prize” you ask? It’s easy! Simply go to Rockpapershotgun.com and answer this simple question: “What is the best science fiction film, and why?” Full details are available on their site and who knows, maybe that second account you always wanted can be yours!