mdemaria: 5000 opteron based computers cannot be put into a tractor trailer and simply plopped down someplace and expect to start computing. adep: why not? and is that total number of physical nodes or processors? if cpus, it would be 625 4-way dual-core nodes s/would/could that would be roughly 20 full racks ...which would fit nicely in a 53' semi trailer. :) mdemaria: which would have an insane amount of power and cooling requirements adep: well of course, that's why said trailer would be parked next to a second truck carrying a genset truck number three would be a fuel tanker to feed that, and finally a large Winnebago to house the op staff. :D FORUM: The time is now 17:00. adep: oh... how much DASD and NAS/SAN disk would these nodes need? mdemaria: 3.5 petabytes adep: ok is that aggregate of all nodes, or network-based? mdemaria: this is a cringely prediction adep: I know mdemaria: doesn't specify adep: but keep feeding me, I want to see if I can do it mdemaria: We're talking about 5000 Opteron processors and 3.5 petabytes of disk storage that can be dropped-off overnight by a tractor-trailer rig. The idea is to plant one of these puppies anywhere Google owns access to fiber, basically turning the entire Internet into a giant processing and storage grid. adep: is this the APC mobile datacenter concept? mdemaria: I don't htink APC fits that much in that space. doesn't the apc center require 2 or 3 trucks? adep: you're right, because they have the genset and HVAC integrated into just one trailer I think they'd only have room for about 12-14 racks I dreamed this concept up myself several years ago mdemaria: how much would that weigh? adep: weight now, or keep working out details? mdemaria: lets say they fit the 5000 opterons in one trailer. forget power/cooling adep: ok I'm still going on my quad-dual platform mdemaria: ok adep: the node would be a 2U I'd rather do 1U but we might need the extra PCI slots mdemaria: you can do a quad 1u? adep: for fiber/iSCSI HBAs hell yeah mdemaria: Let's drop PCI cards out of hte equation for simplicity 625 1U devices, racks and a 2.5PB SAN adep: hang on... that's 32 nodes each rack shitticus... ok I did the math backwards before. need more than 20 total mdemaria: more than 20 racks? If each rack is 19" across, that'd be 31ft for 20 racks adep: I costed the horiz space at 2' each mdemaria: so you can put up to...I'd say you could fit 30 racks in the shipping container adep: remember a trailer is 53' (I wonder how they came up with that size) ok see stop for a minute. if you're just storing them, fitting them into the trailer and not going to RUN them in there... you can do two, three rows of racks I had in mind a single row with the idea they'd run in the trailer mdemaria: I think that's the only way possible. The trailer can't be wide enough to air cool two rows of racks adep: thank you. the machinery is running in the trailer. I need a couple minutes to work out the weight cooling is dead simple in this scenario all you have to do is drop the trailer's sides :D mdemaria: and place it in the high points of utah adep: conceivably no HVAC gear mdemaria: oh yeah, I was ignoring certain apsects of physics for this theorization ignore the cooling Imaging if the cooling broke down for a few moments though. Our machien room can get over 100 within half an hour if the air conditioner is turned off adep: vents dood for the simple reason that you have so much hot exhaust I would not cool/recirculate the exhaust It would make more sense to draw in fresh air always before I can do the weight, I need to figure out how many disks would be needed to reach 2.5 pB the processor nodes would total 14-15 tons. table that, go on to the disk... can we please assume that 2.5pB is total raw storage, not available redundant space? mdemaria: ok adep: thank you :) mdemaria: otherwise it'd be x2 adep: I'm going to spec 2.5" 146gb disks mdemaria: 13107 drives of 200GB I think adep: 7 disks = 1tb 7000 disks = 1pb 14000 = 2pb 17500 = 2.5 these are the smallest physical/largest capacity disks on the market today mdemaria: ok so 17.5K disks fuck man. you'd need a platoon from the active reserves just to swap the failed ones. adep: going for maximum enclosure density I think we can fit 48 disks per 2U I -know- someone does 24/2U, but I think we can double-rack mdemaria: that's 366U needed. adep: ... yep. which means... 9 racks for just disk so I'm up to 29 racks...which is too many for one single row in one trailer ! oh but wait mdemaria: use reaaaaaaaally thin racks made out of titanium adep: ...or taller ones wait a minute for an adjustment 20 racks of 32 would be 640, and we're using 625 15 racks of 42 = 630 ok the racks are stacked FULL now 15 racks of proc nodes + 9 racks of disk = 24 so we're back into the trailer now...we haven't taken infrastructure into account we need some network iron and fiber or SAS or whatever controllers and switches mdemaria: we'd need about 650 network ports adep: we can use 48 port 1U switches those exist right? or at least 48 port modules for BIG switches? let's say 48 port 1U switches 1 switch at the top of each rack in that last U mdemaria: ok adep: then the storage. I have no idea how much fiber gear that'll need mdemaria: one fiber uplink to each rack adep: where's Paras? hehehe mdemaria: than 10GE to storage device adep: oh, iSCSI ok wait mdemaria: you can get a 48+2 1U switch adep: but but don't we need separate physical networks for data and storage? mdemaria: let's not adep: ok let's guess we need one rack for the fiber gear and central BigSwitch mdemaria: ok adep: include in there, the router and CSU oh btw for *damned* sure the nodes are 100% headless no KVMs! hahahahah we're at 25 total racks what is the external width of a 19" rack? 23" ? mdemaria: about adep: let's say 24"...2 feet so 50 feet in the 53 foot trailer done afk, dogging some hots mdemaria: diet coke fizzes funny FORUM: The time is now 18:00. adep: you're right, it does I've been drinking it for 20+ years mdemaria: have you calculated the weight? adep: but I never noticed mdemaria: It fizzes like Guinness adep: I was just calculating proper bun-toasting technique mdemaria: 330 degrees surface temperature. adep: or maybe noticed, but never thought about it Ball Park fat-free franks on Stroehman Potato Rolls. :D these things are really frigging good they're beef pork turkey blend now back to thinking mdemaria: I think we have all the calculations made. We're looking at, if I am recapping correctly: 25 racks 17500 disk drives 25 1U routers adep: 1 pound each disk the 2.5"s are light mdemaria: a bigass switch and 625 1U opteron PCs adep: the proc nodes are 14 tons the disks are 8.75T, round up to 9 then the weight of the ENCLOSURES for the disks need to check scrollback... ok 183 enclosures all aluminum, lattice frames rather than solid panels, to go along with my previous cooling ideas and a supply in there... ok 15 pounds each bare enclosure 2745 pounds disks + enclosures round off to 10 tons even mdemaria: 14 tons for nodes, 10 tons for drives adep: now what do 25 racks weigh? just the racks themselves 100 pounds each? mdemaria: I bet heaver adep: ok 200 that's 5000 nice round number mdemaria: Let me check adep: wait... mdemaria: APC 42U netshelter 295lbs adep: I don't want to use netshelters I want to use all 2-post open frames mdemaria: you have to use 4 post adep: sez who? ok for the disk crates mdemaria: the ass of the PCs will crush themselves adep: no silly the correct way to mount a server in a 2-post is to move the mount point back to the center mdemaria: how many PCs would go in each rack? adep: 42 + 1 switch mdemaria: how much does that weigh? adep: one even ton mdemaria: 2000 pounds? adep: including nuts bolts and cables yes mdemaria: that's the weight capacity of hte netshelter adep: I'm figuring 45 pounds each proc node mdemaria: we'd have to see if the 2 post can support that much weight adep: easily we'll use carbon-fiber alloy :) even using steel it could be done. mdemaria: max load 1500lbs oh wait that's aluminum adep: wait a minute the numbers don't hash earlier I had... OH! we changed from 32 to 42 nodes per rack yes, 42 * 45 = 1890 (45# to include ears & mounts) ten pounds for the switch on the top of the stack and 100 pounds of cabling actually we won't need that much just power and network how much does a power cord weigh? mdemaria: let's add cabling at the end adep: too late I got that 1890 and I'm rounding up to include the cabling is a little but mdemaria: APC 2 post rack - max weight 750lbs adep: that's not carbon-fiber ;) mdemaria: what are we up to without the racks? adep: one even ton per proc rack including: 42 proc nodes, 1 switch, all power and network cabling and four zero-U PDUs for each rack fuck me with 2-post racks, we have nowhere to hang those mdemaria: let's go with 4 post for stability. This thing moves. adep: looks like you're getting netshelters after all mdemaria: so 2200 pounds basically per rack adep: wait netshelters aren't seismic rated they can't absorb/dampen enough shock mdemaria: we'll fill the entire truck with bubblewrap before moving adep: I LIKE IT! mdemaria: and pop each bubble on arrival so we have 2200 pounds in 15 racks for CPUs right? adep: I think we should go up to 2500 mdemaria: which is 16.5 tons adep: we'll take the wheels off the netshelters bolt them all down they won't need to roll out and the wheels are the weak point in the weight rating mdemaria: and 10 tons for disks and more for the infrastructure so around 27-30 tons total or 60,000 pounds. That's a lot of weight. Can that go over bridges? adep: most I've seen some bridges restricted to 30T mdemaria: that's not including the weight of the trailer itself in that number adep: wait ok yeah we're looking at 40T total rig weight mdemaria: yeah that's not marching down main street. adep: want to move the disks to a separate truck? mdemaria: no because you need truck #2 to supply air conditioning and power requirements adep: that would roll over to 3 :) mdemaria: Truck #3 is the cooling van. no wait Truck #2 is power/cooling. Truck #3 is diesel fuel. adep: just roll over one position mdemaria: Feel free to figure out power draw :D adep: I've already started it'll take a bit longer mdemaria: Then ask APC how big of a UPS you'd need adep: UPS gear integrated in genset truck, btw mdemaria: that'd be a big ass battery adep: no, that'd be a metric shitload of batteries mdemaria: shitload and a half adep: but maybe we forego those and just run line conditioners after the generators mdemaria: I dont think UPS are necessary. we'll go with diesel generators adep: nah mdemaria: should recapture energy with wind power. adep: propane mdemaria: 625 servers will generate some air movement adep: solar panels on roof of every truck! hahahah 220V AC power infrastructure? or DC-to-DC ? mdemaria: AC this is suppose to be a google invention. They use commodity adep: ok right power draw for the processor nodes = 412,500 watts ...assuming you need them all on at once actually that's too high 385,000 watts (point of reference: 93Q has a 50,000 watt transmitter.) mdemaria: KGOG Radio adep: WGOG. we're in NY :) the W/K line is drawn at the Mississippi river. figure 180,000 watts for the disk crates working on infrastructure... FORUM: The time is now 19:00. adep: 22,000 watts should about do it actually no that's more than enough so mdemaria: 587,000 watts adep: I'll say the generator load will be 560 kilowatts that's constant in-use mdemaria: 30,000 volts at 20A adep: my theoretical disk crates each have 800 watt power supplies which is a lot until you remember that each crate is 7tb :D adep>