Company

CIS Hollywood Buffs Up Its Infrastructure With BlueArc Storage and Force10’s Core Switch

Staying ahead of the performance curve is always a challenge – even more so for visual effects house CIS Hollywood, which creates 500 gigabytes to one terabyte of data daily. For CIS Hollywood’s IT team, the challenge is to maintain an infrastructure that can keep up with the increasing complexity of special effects projects as well as the company’s growth.

Systems Manager Matt Ashton is clear about his goals for the IT infrastructure: It must operate at line rate, with almost no latency under full load. Through careful monitoring and planning, Ashton was able to time storage system and network core overhauls to keep ahead of the workflow requirements of artists, rendering nodes and other systems on the LAN.

That planning continues to pay off. With consolidated, high-performance storage based on BlueArc’s Titan 2200 Storage System and a new core based on Force10 Networks’ TeraScale E1200 switch/router, CIS Hollywood is able to handle up to a dozen projects simultaneously. In 2006, the new infrastructure enabled CIS Hollywood to work on four major films in parallel: X-Men: The Last Stand, Poseidon, The Break-Up, and The Fast and the Furious: Tokyo Drift.

Ashton recognized that storage and network performance are both crucial – a bottleneck in either can affect workflow. By selecting BlueArc and Force10 systems, he’s found a cost-effective way to accommodate content growth of roughly 30 percent per year with no hiccups in workflow.

Growing in All Directions

In the four years he’s been with CIS Hollywood, Ashton has watched the business expand and change. The number of employees jumped six fold, with each artist maintaining multiple systems on their desktop. Likewise, the number of render nodes now exceeds 300.

"We have more business going at the same time," says Ashton. "Instead of working on one or two shows, we’re working on a half dozen to a dozen jobs at a time." And the nature of the content has changed, becoming larger and more complex.

"One of the biggest things in the industry right now is the move toward 4K frames," Ashton notes. That means that "a show that previously took 12.6 megabytes per frame now is going to take more than 50, quadrupling the storage space required." The network also takes a hit. The file transfers for opening, saving and rendering data are much larger, requiring more network bandwidth. "Rendering takes that much longer as well, and the artists have to have continuous throughput," adds Ashton.

Two years ago, only about five percent of the films CIS Hollywood worked on used high-resolution 4K frames versus 2K frames. Today that percentage is closer to 50, Ashton says. In addition, the effects being produced, such as fluid simulation and crowd simulation, place significant demands on the IT infrastructure. "With fluid simulations, millions of small files are generated and then have to be chomped on by the render nodes," he says.

"Complexity’s gone up, the number of shots going through at any given time has gone up, the number of people working has gone up – everything’s gone up," says Ashton. To keep pace with the growth, he began looking for a new storage solution and then a core switch as the company maxed out first one system then the other.

 

Streamlining Storage

With storage capacity running low and latency beginning to creep up, Ashton considered various options, including purchasing a fourth network-attached storage (NAS) appliance from his previous supplier. The existing three legacy NAS devices each housed three terabytes of data, a cap Ashton placed on them to ensure they weren’t oversubscribed. The fourth unit he considered would add another six terabytes but would mean managing four units and paying software licensing and support costs on four devices.

In contrast, BlueArc’s Titan Storage System could provide CIS Hollywood 12 terabytes of storage on a single centralized platform for less than twice the cost of the legacy NAS vendor’s six terabyte platform. "The cost of adding one more NAS node was more than half way towards the price of the BlueArc. The support costs of the legacy system were starting to become prohibitive as well," Ashton says. "With the amount of storage we get with the BlueArc, economically it seemed like a pretty good choice."

CIS Hollywood recently upgraded to 16 terabytes of storage on BlueArc’s Titan system, ensuring there’s room to grow. Not only did Titan allow Ashton to consolidate all data on a single storage system, but the Titan’s hardware-based design delivers higher performance than the legacy NAS appliances, eliminating the need for so many filers. "The BlueArc is very fast, noticeably fast," Ashton says.

BlueArc’s storage systems also feature a modular architecture that allows customers to scale storage capacity without downtime. In addition, the Titan Server’s modularity enabled CIS Hollywood to increase performance with a simple blade upgrade, eliminating a forklift upgrade or new hardware and licensing costs for additional filers.

With Titan’s potential to achieve up to 800 MB/s of throughput, over 200,000 input-output operations per second, and up to 512 terabytes of capacity, CIS Hollywood has room to scale its storage infrastructure to keep pace with growth in staff, projects and graphics resolution. Titan provides high levels of performance even during project crunch times, when render farm and artist access is at its peak. CIS Hollywood also has the option of adding tier two Serial ATA storage, for more cost-effective storage for completed works or lower performance needs.

BlueArc’s combination of performance and scalability are crucial for giving CIS Hollywood the flexibility it needs to accommodate fluctuating work loads and to take on last-minute overflow jobs from other effects studios. "When we add another person, they can only work on so many shots," says Ashton. "But what type of shots are they? What type of work are we getting? It’s difficult to forecast that I’m going to need four more terabytes – it depends on the work."

Expanding the Core

To maximize performance, Ashton designed his network so that all devices, including render nodes and artists’ workstations, connect directly to the core switch. Given its growth, the company inevitably maxed out its legacy core switch. "The switch was fully populated," Ashton says. "Once the core switch gets full, then you have to start adding edge switches and you start introducing latency and complexity into the network. I didn’t want to go that route."

Ashton began searching for a high-capacity, line-rate, non-blocking core device that could handle spikes in traffic as workflow demanded and maintain predictably high performance under these wildly varying loads. He also wanted a platform with line-rate 10 GbE capability and support for numerous fiber ports. Having only four fiber ports "was a huge limitation" of the legacy core switch, he says.

He evaluated equipment from several vendors, including his legacy vendor, Force10 Networks and two other networking companies. At the time, the legacy vendor’s 48-port card didn’t support direct attachment, but rather required the use of adapters. "That adds another point of failure and I don’t like that," Ashton says. In contrast, Force10 offered 48-port cards with direct attachment. 

In addition, the legacy vendor’s newer, high-end platforms "don’t have the same kind of bandwidth that Force10 does, so there was another plus for the Force10 core switch/router," says Ashton. He adds that Force10’s "reputation as the 10 gig market leader was a big factor" in leading him to evaluate the company’s products.

He eliminated one major networking vendor because their gear was too costly, and nixed another switch maker for lagging overall. He opted for Force10’s E1200, which features a 1.68 Tbps non-blocking switch fabric, a forwarding capacity of 1 billion packets per second, and 14 line-card slots that can be configured with a mix of up to 672 GbE or 56 10 GbE line-rate, non-blocking ports.

Currently, Ashton has the E1200 half populated, with six slots configured with 48-port GbE cards and one slot configured with gigabit interface converter (GBIC) ports. As he moves to implement 10 GbE, "I’m going to stick with the line rate stuff," he says. "We try to push as much as we can per port. It’s unacceptable for it to be oversubscribed and then have it drop out."

Although Ashton wants to ensure maximum uptime, building a redundant core is cost prohibitive. "I couldn’t get away with that. So I opted for the best chassis I could get," he says, and equipped it with redundant power supplies, management and routing modules, giving CIS Hollywood multiple levels of physical redundancy to ensure no single point of failure.

 

Capacity to Spare

In the demanding world of feature films where the workflow can be unpredictable, having the right infrastructure in place is key to success. By implementing systems from BlueArc and Force10, Ashton has ensured CIS Hollywood has the storage and network capacity at the performance levels it needs to deliver quality effects on deadline.

In addition, CIS Hollywood has capacity to spare, giving it the flexibility to take on last minute projects from other studios even as it grows its own business. When management says, "By the way, we’ve got four more shows and two more people starting tomorrow," Ashton is confident the infrastructure can handle it. His goal is to stay ahead of the workflow requirements – with Force10 and BlueArc he’s been able to do just that.

And "not having to worry about the equipment is a big thing," he notes. "When you don’t have confidence in it, in the back of your mind you’re always worrying – is it going to fail?" Ashton knows that with BlueArc and Force10 the show will go on.

back to top >