Next Generation System DesignChallenges and OpportunitiesBy Derek TsaiSun Microsystems10/27/2001Copyright Sun Microsystems, EPEP, Derek TsaiPage 1

ContentsSun’s Mid range to High end servers and marketKeeping the scores metrics of successMap out the value chainIdentify the BottlenecksKey challengesKey opportunitiesConclusions10/27/2001Copyright Sun Microsystems, EPEP, Derek TsaiPage 2

Sun’s Mid range to High endservers: current line upSun fire 3800Up to 8 CPUsSun fire 4800Up to 12 CPUsSun fire 6800Up to 24 CPUs#1 in U.S. Server Revenue (21%)*#1 in Worldwide Unix Server (35%)** IDC 3/0110/27/2001Copyright Sun Microsystems, EPEP, Derek TsaiSun fire 15000Up to 106 CPUsPage 3

Server Markets Fueling worldwide productivity growthMarkets SegmentsIndustriesManufacturingTelcoService Entertainment/Media10/27/2001CRMHPCTechnical DevelopmentCollaborativeData Warehouse/Bus. Intel.Web ServicesPortal ComputingE MarketplacesMobileOLTPERPSupply Chain ManagementCopyright Sun Microsystems, EPEP, Derek TsaiPage 4

Server System Design Challenges –From 30,000 ftCostsCompetitive forces (dot com burst, Wintel servers)Perception of Benefits (Wintel servers, bad economy)Thermal/Power management:From the box out (2x to 3x of power density/generation)From the data center in (Max’ed out on cooling, power shortage)Scaling Up of performanceStressing interconnect bandwidthMore modules 0 more interconnectsBoosting RAS (Reliability, Availability, Serviceability)Complexity , Heat , Bandwidth , Interconnects 10/27/2001Copyright Sun Microsystems, EPEP, Derek TsaiPage 5

Keeping the scores metrics of successWhy keep scores/metrics?Too easy to over design and add unnecessarycostsDe mystify the signal integrity black magicCapture Signal Integrity Engineers’ value add(not a prohibitor)Know the trend to reduce risksSun’s Six sigma initiative (Sun Sigma)10/27/2001Copyright Sun Microsystems, EPEP, Derek TsaiPage 6

Keeping the scores metrics of success (Cont’ed)Possible metrics:I/O BandwidthI/O power consumptionI/O noise marginPower noisePin/route densityPower density10/27/2001Copyright Sun Microsystems, EPEP, Derek TsaiPage 7

Keeping the scores metrics of success (Cont’ed)Ultimate benefit for systemsI/O bandwidth/throughputLatency (dominated by placement, architecture, andspeed of light)Costs (component and implementation)power/thermalNoise (electrical and Electromagnetic)Packaging (pin/route density, material, & etc.)Reliability10/27/2001Copyright Sun Microsystems, EPEP, Derek TsaiPage 8

Keeping the scores metrics of success (Cont’ed)Proposed metrics for SI/Packaging Engineers:I/O bandwidth / pin / Watt consumed / ANDMeeting spec’ed reliabilityMeeting EMI regulationHaving sufficient noise margin for all component andmanufacturing process corners % Operating Freq Margin / AND same conditions as above10/27/2001Copyright Sun Microsystems, EPEP, Derek TsaiPage 9

Keeping the scores metrics of success (Cont’ed)Sun’s Sparc CPU Speed & I/O BandwidthCPU Core SpeedMemory BW/Pin/W1000010000CPU Core FreqFitted CAGR: 45%/yr10001000MB/Pin/WCPU Speed (MHz)Memory BW MB/Pin/WFitted CAGR: 020011002002Year10/27/2001Copyright Sun Microsystems, EPEP, Derek TsaiPage 10

Keeping the scores metrics of success (Cont’ed)Sun’s Sparc Memory I/O BandwidthMemory BW/Pin10000Fitted CAGR: 40%/yrMemory BW/Pin/WFitted CAGR: 65%/yr101000Memory BW/PinMemory 001MB/Pin/WMemory Bandwidth/Pin (MB/s)1001002002Year10/27/2001Copyright Sun Microsystems, EPEP, Derek TsaiPage 11

Keeping the scores metrics of success (Cont’ed)I/O Standards SurveyInfiniband x12Aggregate Bandwidth (MBytes/s)10000Infiniband x4InfinibandFibrechannelx11000 FC AL10010Ethernet 1000baseTIEEE 1394Ethernet 100baseTEthernet 10baseTWide Ultra3 SCSIWide Ultra2 SCSIUltra3 SCSIUltra2Ultra SCSIIDEFast WideIDE SCSISCSI 2SCSI 1ECP/EPP ParallelUltra Wide SCSIUSB2Ethernet10base21USB1Std Parallel0.1RS2320.01010/27/20014812I/O Width16Copyright Sun Microsystems, EPEP, Derek Tsai2024Page 12

Mapping the value chain ofsystem designWhy map the value chain?Determine the bottleneck and invest to removeBottlenecks move around for each generationReduce the risksReduce the costs of over designBecause it’s a SYSTEM10/27/2001Copyright Sun Microsystems, EPEP, Derek TsaiPage 13

Mapping the value chain ofsystem designSystem Level: Signal PathPCBPCBChip PackageI/O AreaChip PackageConnectorConnectorCoreI/O AreaCoreCableI/O Cell, ESD , Chip package, PCB trace, vias, connector, transceivers, cables10/27/2001Copyright Sun Microsystems, EPEP, Derek TsaiPage 14

Identifying the bottlenecksSignal path bottlenecksI/O Cell Design: from Global Sync to Source SyncSimultaneous switching I/O noiseESD loadPackage tracePackage/PCB interfacePCB trace loss and routabilityPCB viasSignal referenceTransceiversCables (copper and optic)10/27/2001Copyright Sun Microsystems, EPEP, Derek TsaiPage 15

Mapping the value chain ofsystem design (Cont’ed)System Level: Power PathPCBPCBChip PackageI/O AreaChip PackageConnectorConnectorCoreI/O AreaCoreCablePowerConnectorVRMPower connector, VRM, Capacitors, PCB planes, on package caps, on die caps10/27/2001Copyright Sun Microsystems, EPEP, Derek TsaiPage 16

Identifying the bottlenecks(Cont’ed)Power path bottlenecks:Power connector inductance & resistanceVRM – power transient, densityPCB Power – spreading inductancePackage PowerCore Power – power transient, package resonanceI/O Power – return reference, SSNPower across domains cabling10/27/2001Copyright Sun Microsystems, EPEP, Derek TsaiPage 17

Key ChallengesInterconnect densityMechanical:Qualification of connectors, pcb and packagesFinding the failure modes (too many interfaces)Assembly and handlingRouting and pin escapesLead freeElectrical:Maintaining the signal quality: lossiness, via loss, crosstalk,return paths, SSN, Bit Error Rate, and etc.EMIDiagnostics and testabilityObservability: design for debug and measurementEconomics: total costs of ownership/design10/27/2001Copyright Sun Microsystems, EPEP, Derek TsaiPage 18

Key Challenges (Cont’ed)Power and CoolingDelivering power:Power architecture (staging/budgeting) criticalOptimal use of decoupling capacitors: VRM, board level, package level, and chip level.Availability and accuracy of capacitor modelsThermal:Efficient use of power – justify the demandsE star mode10/27/2001Copyright Sun Microsystems, EPEP, Derek TsaiPage 19

Key Challenges (Cont’ed)Design ComplexityModel availability and accuracy:SPICE vs. IBIS vs. others: methodology wantedFrequency dependent interconnect modelsResource drain to provide accurate modelsSimulation:HSPICE vs. SpecctraQuest vs. XTKWhat to look for: timing, error rate, eye diagramWhat to include in simulation: too many variables,information overload determine what are essential.Tedious must deploy computer tools/scripts to excel10/27/2001Copyright Sun Microsystems, EPEP, Derek TsaiPage 20

Key Challenges (Cont’ed)SI Engineering ResourcesLack of hands on debug experience: video gameculture creates lots of naïve engineers lack "instincts"Specialized to be competent, but not toospecialized to be practical insufficient "system" viewNo glory: hard to extract our value add – need goodsuccess metricsCase studies needed in teaching SI no lesscomplex than business cases10/27/2001Copyright Sun Microsystems, EPEP, Derek TsaiPage 21

Key OpportunitiesMove away from Black Magic to Standard SIdesign and test methodology: put aside the egoand show me the dataIndustry wide Simulation methodology:Component suppliers to provide accurate models, workingwith tool providersIndustry tool certification body (JEDEC)supplying benchmark test cases like TPC for databaseSpecialized SI curriculum with lots of real life casestudies provided by industry10/27/2001Copyright Sun Microsystems, EPEP, Derek TsaiPage 22

ConclusionsInterconnect/Packaging is the bottleneck now andSystem is as good as the bottleneck: must solve it asa system to avoid large and risk costsSome of Sun’s challenges are shared by theindustry:Better metrics of success common language/goalJoint solution can enlarge the pie for all and start withproviding/sharing accurate models, andOpen interface to SI tools allows users to tackle nextdesign challenges10/27/2001Copyright Sun Microsystems, EPEP, Derek TsaiPage 23