Potential servers

Potential high performance per watt, cheap servers.

Super-low power CPUs

Atom C3000

Atom C3000 aka Denverton. Supermicro is making some, list, Gigabyte too. Nothing available as of 2017-10-02. I asked Interpro about prices and availablity for the following:

$280 A2SDi-4C-HLN4F   4-core , 8 SATA, 4x1Gb
$427 A2SDi-8C-HLN4F   8-core, 12 SATA, 4x1Gb
$603 A2SDi-12C-HLN4F 12-core, 12 SATA, 4x1Gb
$714 A2SDi-16C-HLN4F 16-core, 12 SATA, 4x1Gb
$?   A2SDi-TP8F      12-core, 4 SATA, 4x10gb, 2x10Gb, 2xSFP+, SO-DIMM
$833 A2SDi-H-TP4F    16-core, 12 SATA, 2x10Gb, 2xSFP+
$479 A2SDi-H-TF       8-core, 12 SATA, 2x10Gb

Some of these boards have Intel Quick Assist Technology aka QAT, which is hardware crypto stuff. The drivers are FOSS, but the devices require loading non-free binary blobs (which are in firmware-misc-nonfree). Scary stuff. I don’t know if there is a way to use the system that ensures QAT isn’t loaded, we need to investigate more.

Atom C2000

Atom C2000, 8-core, 4 SODIMM slots (25GB/s memory bandwidth!), uses similar architecture to haswell, uses Xeon cores, 20W, includes 4x 1gb/2.5gb NICs, cheap, awesome. Basically the main downside compared to haswell is lack of additional channels to talk to pci-e slots, but it makes up for that by having most of the stuff we want onboard (sata, usb3, nics). So for systems where we won’t be adding additional SATA controllers or NICs, it is a cheaper alternative. Also some vendors are including interesting things on their boards like extra sata controllers.

Two versions: Avoton and Rangeley. Rangeley is the same as Avoton plus,

Rangeley is a tweak of Avoton that turns on the QuickAssist Technology (QAT)
accelerator on the chip, which hooks into Intel's Data Plane Development Kit
for network gear makers to juice AES, DES/3DES, Kasumi, RC4, and Snow3G
ciphers, MD5, SHA1, SHA2, and AES-XCBC authentication, and Diffie-Hellman,
RSA, DSA, and ECC public key encryption. This QAT coprocessor can process
ciphers at 10Gb/sec. And by the way, not all of the Rangeley chips will have
this QAT accelerator activated; to be precise, only four out of the eight
SKUs will. The reason for this is that the QAT accelerator is a controlled
substance and the US government has export controls on it.

We should be suspicious of built-in crypto, it’s unclear if any of it could be used safely.

Atom C2000 boards

Supermicro
Two boards, both are octa-core, 4xSODIMM, 6xSATA, 4xNIC, 1xIPMI NIC
A1SAi-2750F (Avoton) $359.65 @ interpromicro.com, Review
A1SRi-2758F (Rangeley) $359.65 @ interpromicro.com

This combo is mainboard+cpu+heatsink. Compared to our haswell config this is $360 vs. $498.25 ($167.50+$311.50+$19.25). The ECC SODIMMs are more expensive ($95 vs $71.50 for an 8gb) so some of that savings is lost.

ASRock
Two boards coming in Nov/Dec, forum post. Both have 4xDIMM (full, not SODIMM), 2xNIC, IPMI, 12xSATA!!!

Atom C2000 sample configs

Supermicro 8-core Minimal config

Board/CPU A1SRi-2758F (Rangeley) $360
Case 510T-203B $145
RAM 2×4gb DDR3-1600 ECC SODIMM $110
SSDs 2×120gb Samsung 840EVO $198
Total $813

Supermicro 8-core Decent config

Board/CPU A1SRi-2758F (Rangeley) $360
Case 510T-203B $145
RAM 4×8gb DDR3-1600 ECC SODIMM $380
SSDs 2×240gb Samsung 840EVO $360
Total $1245

ASRock 8-core Minimal config

Board/CPU C2750D4I $373
Case 510T-203B $145
RAM 2×4gb DDR3-1600 ECC DIMM $100
SSDs 2×120gb Samsung 840EVO $198
Total $816

ASRock 8-core Decent config

Board/CPU C2750D4I $373
Case 510T-203B $145
RAM 4×8gb DDR3-1600 ECC DIMM $420
SSDs 2×240gb Samsung 840EVO $360
Total $1298

ASRock 4-core Minimal config

Board/CPU C2550D4I $290
Case 510T-203B $145
RAM 2×4gb DDR3-1600 ECC DIMM $100
SSDs 2×120gb Samsung 840EVO $198
Total $733

ASRock 4-core Decent config

Board/CPU C2550D4I $290
Case 510T-203B $145
RAM 4×8gb DDR3-1600 ECC DIMM $420
SSDs 2×240gb Samsung 840EVO $360
Total $1215

Celeron J1900

The Celeron J1900, a Bay Trail-D model, is also part of the Silvermont CPU family (like the Avoton above). It has similar specs to the Avoton, slightly lower clock, lower power (10W vs 14W), integrated graphics, and only supports 8gb of RAM. But it is MUCH cheaper since it’s high-volume, low-end desktop market. They are quad-core, 2M cache, 64bit and support hardware virtualization. Board+CPU combos are ~$55-80. They won’t have server features like IPMI and high end PCIe, but some have serial ports (or serial headers). They may not have the ability to do serial BIOS, but there is some support for Bay Trail in coreboot, it may be possible with some work to get coreboot working to get it.

NOTE: we haven’t tested any of these yet, but they are cheap enough we might try. If you try it, let us know.

Board Price Form RAM SATA Serial NICs Other
ASRock Q1900-ITX $76 mITX 2xSODIMM 2×3, 2×6 header 1 1xMiniPCIe,1xPCIe
ASRock Q1900DC-ITX $76 mITX 2xSODIMM 2×3, 2×6 header 1 1xMiniPCIe,1xPCIe, DC version of above
ASRock Q1900TM-ITX $118 mITX 2xSODIMM 2×3, 2×6 header 1 1xMiniPCIe,1xPCIe, DC+thin version of above
ASRock Q1900M PRO3 $69 mATX 2xDIMM 2×3 yes 1 2xPCIe,2xPCI
ASRock Q1900M $70 mATX 2xDIMM 2×3 header 1 3xPCIe
ASRock Q1900B-ITX $70 mITX 2xSODIMM 2×3 yes 1 1xPCIe
ASUS J1900I-C $77 mITX 2xSODIMM 2×3 yes 1 1xPCIe
ECS BAT-I (V1.2) $55 mITX 1xSODIMM 2×3 header 1 1xMiniPCIe
ECS BAT-I2 (V1.0) $? mITX 1xSODIMM 2×3 header 1 1xMiniPCIe
ECS BAT-I3 (V1.0) $? mITX 1xSODIMM 1×3 yes 2 1xMiniPCIe,1xMiniPCIe(short),1xPCI,LVDS
ECS BAT-TI (V1.0) $? mITX 1xSODIMM 2×3 no 1 1xMiniPCIe,LVDS
ECS BAT-TI2 (V1.0) $? mITX 1xSODIMM 2×3 header? 1 1xMiniPCIe,LVDS,DC
Foxconn D190S $65 mITX 1xSODIMM 2×3 yes 1 1xMiniPCIe,1xPCIe
GIGABYTE GA-J1900N-D3V $85 mITX 2xSODIMM 2×3 yes 2 1xMiniPCIe,1xPCI
Supermicro X10SBA $162.05 mITX 2xSODIMM 2×6,4×3 headers 2 1xMiniPCIe,1xPCI
Supermicro X10SBA-L $136.50 mITX 2xSODIMM 2×3 headers 2

Celeron J1900 configs

  • DDR3-1333 2×4gb are around $75-80, for both SO-DIMM and DIMM.
  • some of the boards support mSATA
  • some support 2xSATA, some 4xSATA, so maybe 2xSSD and 2xHDD → $260-400
  • they are all intended to be in desktop cases, but should all be low profile enough to fit in a 1U case if you remove the I/O shield and the case has ATX mounting holes and ATX power connector. Front panel LED/buttons might be tricky

Xeon D configs

Supermicro Xeon D-2100 2018 products

Model CPU Watts Price Notes
X11SDV-4C-TLN2F D-2123IT, 4C8T/8MB/2.2/3.0 60W $545 2 10Gb-T
X11SDV-8C-TLN2F D-2141I, 8C16T/11MB/2.2/3.0 65W $919.50 2 10Gb-T
X11SDV-12C-TLN2F D-2166NT, 12C24T/17MB/2.0/3.0 85W $1505 2 10Gb-T
X11SDV-16C-TLN2F D-2183IT, 16C32T/22MB/2.2/3.0 100W $2425 2 10Gb-T
X11SDV-8C-TP8F D-2146NT, 8C16T/11MB/2.3/3.0 80W $1125 4 GbE, 2 10Gb-T, 2 10G SFP+
X11SDV-12C-TP8F D-2166NT, 12C24T/17MB/2.0/3.0 85W $1579.50 4 GbE, 2 10Gb-T, 2 10G SFP+
X11SDV-16C-TP8F D-2183IT, 16C32T/22MB/2.2/3.0 100W $2495 4 GbE, 2 10Gb-T, 2 10G SFP+

(The X11SDV-8C+-TLN2F and X11SDV-16C+-TLN2F have a CPU fan for use in cases that don’t depend on case fans, but are otherwise identical to the non-plus models)

Older D-1450 listed in table below.

Previous Generation Intel Atom

The current shipping Atom CPUs are code named “Pineview” (platform is “Pinetrail”) and they are based on a 45nm process.
The next generation of Atom CPUs are code named “Cedarview” (platform is "Cedartrail) and are based on a 32nm process. They aren’t expected to ship until 2H2011. I have read two different rumors

  1. They will have twice the performance of the current
  2. They will only be about 5-10% faster, but their power usage will be way lower. One article said they are aiming to have netbook type devices where the entire thing only needs 10W!

If the latter is true (and I think it’s more likely) then it might not make sense to hold out for the newer ones. The pineview cpu’s are 13W, already nearly negligible for a colo server power budget. (maybe this will push us more towards the low power quad core cpus, since they are likely to drop in power with a process shrink and we’d get more performance)

Atom Pineview

The next version Atom will be: 2013, code name Silvermont, 22 nm

Previous generation atom boards

Atom D525 based
X7SPE-HF – $213.25 from Interpro
X7SPE-H – $181.55 from Interpro

Atom D510 based (older, but more available)
X7SPA-HF ~$180 (includes CPU)
X7SPA-H – doesn’t have the IPMI stuff and uses lower power Intel video instead of Matrox

hotplug options

while supermicro still doesn’t make any Atom based 1U chassis with hot-plug disks they do make this 1U chassis with 2 2.5" hot-plug slots and a 200W power supply
SC510T-200B ~$150

I emailed supermicro to ask if these were compatible (you have to check, because 1U servers can’t use a ATX style I/O port shield, it’s too tall for 1U) and they said they were. That mainboard has 2 ram slots and uses “non-ECC DDR2 667MHz(D510)/800MHz(D525) SO-DIMM”s, 2gb per slot, 4gb total.

fixed drives options

SYS-5015A-EHF – D525 based – $363.50 from Interpro
SYS-5015A-PHF – D510 based

These systems can only hold 1 3.5" drive or 2 2.5" drives, and they are not hot plug. The boards have 6 SATAII (aka SATA300, or 3.0Gb/s) ports.

AMD Based

(updated 2014/08/30)
AMD is launching an arm64 CPU that looks pretty awesome for our needs.

  • AMD Opteron A1100 codenamed “Seattle”, 64-bit ARMv8
    • Up to 8 ARM Cortex-A57 cores (64-bit)
    • Dual DDR3 or DDR4 DRAM channels (4 slots possible)
    • 128GB maximum memory
    • Dual 10G Ethernet onboard
    • 8 lanes of PCI-Express 3.0
    • 8x SATA 3 ports integrated
    • Integrated crypto and compression acceleration
    • Integrated management functionality
    • 28nm process initially produced by GlobalFoundries.
  • Specs
  • Comparison to Intel Avoton
  • The Register article
  • AMD Dev kit
  • Group in Debian hoping to produce an ARM laptop using it, by making a board that fits in an existing Lenovo X-class case (and uses it’s screens/batteries/modules/etc) www.vero-apparatus.com

Low power CPUs

This table is focused on getting the best CPU for amount of power and cost and assumes RAM and I/O will be sufficient. If you need better memory bandwidth and/or more I/O slot bandwidth, you’re going to have to pay the premium for the fancy CPUs (that’s why they cost more).

Mainly these are “server” CPUs because they have more cache and memory channels.

AMD Server CPUs

Epyc2 7002 Full chart at wikipedia

Model Clock Turbo Cores (Threads) Cache Cache/ Core TDP TDP/ Core Cost Cost/ Core Cost/ MB Notes
7252 3.1 3.2 8/16 64 8 120W 15W $570 $71.25 $8.91
7262 3.2 3.4 8/16 128 16 155W 19.345W $680 $85 $5.31 Cheapest cache
7272 2.9 3.2 12/24 64 5.33 120W 10W $740 $61.67 $11.56 2nd cheapest cores, 3rd cheapest cache
7282 2.8 3.2 16/32 64 4 120W 7.5W $770 $48.13 $12.03 Cheapest cores
7302 3.0 3.3 16/32 128 8 155W 9.68W $1100 $68.75 $8.59 2nd cheapest cache
7352 2.3 3.2 24/48 128 5.33 155W 6.46W $1510 $62.92 $11.80
7402 2.8 3.35 24/48 128 5.33 180W 7.5W $1990 $82.92 $15.55
7452 2.35 3.35 32/64 128 4 155W 4.84W $2260 $70.63 $17.66 Lowest W/core
7502 2.5 3.35 32/64 128 4 180W 5.63W $2900 $90.62 $22.66 2nd lowest W/core
7542 2.9 3.4 32/64 128 4 225W 7.03W $3710 $115.94 $28.98

Epyc1 7000 – no longer interesting

Epyc 3000 – embedded so cpu comes with mainboard Full chart at wikipedia

Vendor Model CPU Cores/Threads Cache TDP Price
Supermicro M11SDV-4CT-LN4F 3101 4/4 8 35W $430
Supermicro M11SDV-8CT-LN4F 3201 8/8 16 30W $526
Supermicro M11SDV-8C-LN4F 3251 8/16 16 50W $690
Supermicro M11SDV-8C-LN4F 3251 8/16 16 50W $690
ASRock EPYC3101D4I-2T 3101 4/4 8 35W $600
ASRock EPYC3251D4I-2T 3251 8/16 16 50W $900

Intel Server CPUs

Model Family Socket Clock
Turbo Cores (Threads) Cache Cache/ Core TDP TDP/ Core RAM Cost Cost/ Core Cost/ MB CPUMark CPUmark/$ CPUmark/$/W Replacement Notes
1280Lv6 Kabylake LGA1151 3.9GHz 4.2GHz 4(8) 8mb 2mb 72W 11.25W DDR4-2400(4 DIMMs) $570 $71.25 $71.25 ? ? ?
1270Lv6 3.8GHz 4.2GHz $354 $44.25 $44.25 ? ? ?
1240Lv6 3.7GHz 4.1GHz $340 $42.5 $42.5 ? ? ?
1230Lv6 3.5GHz 3.9GHz $273 $34.13 $34.13 ? ? ?
1220Lv6 3.0GHz 3.5GHz 4(4) $214 $26.75 $26.75 ? ? ?
1260Lv5 Skylake LGA1151 2.9GHz 3.9GHz 4(8) 8mb 2mb 45W 11.25W DDR4-2133(4 DIMMs) $330 $82.5 $41.25 ? ? ? Current Best 4(8)
1240Lv5 2.1GHz 3.2GHz 25W 6.25W $307 $76.75 $38.38 ? ? ?
1235Lv5 2GHz 3GHz $282 $70.50 $35.25 ? ? ? Has an “HD P350” GPU
D-1540 Broadwell DE SoC 2.0GHz 2.6GHz 8(16) 12mb 1.5mb 45W 5.6W 2x DDR4-2133(8 DIMMs) $796.50 $99.56 $66.38 ? ? ? Current Best 8(16),price includes X10SDV mainboard
2650lv3 Haswell EP LGA2011v3 1.8GHz ? 12(24) 30mb 2.5mb 65W 5.42W 8x DDR4-2133(16 DIMMs) $1429.50 $119.13 ? ? ? ? ? ?
2648lv3 1.8GHz 75W 6.25W $1597.50 $133.13
2628lv3 2.0GHz 10(20) 25mb 75W 7.5W 4x DDR4-1866(8 DIMMs) $1329.50 $132.95
2630lv3 1.8GHz 8(16) 20mb 55W 6.875W $659.65 $82.46
2618lv3 2.3GHz 75W 9.375W $758.50 $94.82
2608lv3 2.0GHz 6(12) 15mb 52W 8.67W $431.5 $71.92
1265LV3 Haswell LGA1150 2.5GHz 3.7GHz 4(8) 9MB 2.25MB 45W 11.25W 2x DDR3-1600 $311.50@interpro(check) $77.88 $34.61 ? Current Best 4(8)
1230LV3 1.8GHz 2.8GHz 4(8) 9MB 2.25MB 45W 11.25W $265.15@interpro(check) $66.29 $29.46 ?
1220LV3 1.6GHz 2.3GHz 2(4) 4.5MB 2.25MB 16W 8W $203.75@interpro(check) $101.88 $45.27 ?
1265LV2 Ivy Bridge LGA1155 2.5GHz 3.5GHz 4(8) 9MB 2.25MB 45W 11.25W $328(check) $82 $36.44 8823 26.90 0.60 Current Best 4(8) (no ECC)
1220LV2 2.2GHz 3.5GHz 2(4) 3.5MB 1.75MB 17W 8.5W $226(check) $113 $64.57 6746 29.85 1.76 Current Best 2(4)
2650L Sandy Bridge LGA2011(for dual socket) 1.8GHz 2.3GHz 8(16) 20MB 2.5MB 70W 8.75W 4x DDR3-1333 $1140(check) $142.50 $57 14308(dual) 6.28 0.09
2630L 2.0GHz 2.5GHz 6(12) 15MB 2.5MB 60W 10W $689(check) $114.83 $45.93 Current Best 6(12)
2450L LGA1356(for dual socket) 1.8GHz 8(16) 20MB 2.5MB 70W 8.75W 3x DDR3-1333 $1282(check) $160.25 $64.10 2650L
2430L 2.0GHz 6(12) 15MB 2.5MB 60W 10W $767(check) $127.83 $51.13 2640L
1260L LGA1155 2.4GHz 3.3GHz 4(8) 8MB 2MB 45W 11.24W 2x DDR3-1333 $312(check) $78 $39 6502 20.84 0.46 1265LV2
1220L 2.2GHz 3.4GHz 2(4) 3MB 1.5MB 20W 10W $203(check) $101.50 $67.67 1220LV2
L5630 Westmere LGA1366 2.13GHz 4(8) 12MB 3MB 40W 10W 3x DDR3-1066 $776(check) $194 $64.67 1265LV2(-1 RAM channel)
L5618 1.86GHz 4(8) 12MB 3MB 40W 10W $476(check) $119 $39.67 1265LV2(-1 RAM channel)
L3406 LGA1156 2.26GHz 2(4) 4MB 2MB 30W 15W 2x DDR3-1066 $213(check) $106.5 $53.25 1220LV2
L3403 2.0GHz 2(4) 4MB 2MB 30W 15W $?(check) $? $? 1220LV2 can’t find
L7555 Nehalem LGA1567 1.86GHz 8(16) 24MB 3MB 95W 11.875W 4x DDR3-1066 $3950(check) $493.75 $164.58 2650L sucks
L7545 1.86GHz 6(12) 18MB 3MB 105W 17.5W $1550(check) $258.33 $86.11 2630L sucks
L5530 LGA1366 2.4GHz 4(8) 8MB 2MB 60W 15W 3x DDR3-1066 $800(check) $200 $100 1265LV2(-1 RAM channel) sucks
L5520 2.26GHz 4(8) 8MB 2MB 60W 15W $270(check) $67.50 $33.75
L5518 2.13GHz 4(8) 8MB 2MB 60W 15W $370(check) $92.50 $46.25 1265LV2(-1 RAM channel) sucks
L5506 2.13GHz 4(8) 4MB 1MB 60W 15W $240(check) $60 $60 sucks
L5508 2.0GHz 2(4) 8MB 2MB 38W 9.5W $440(check) $220 $55 1220LV2(-1 RAM channel)
L3426 LGA1156 1.86GHz 4(8) 8MB 2MB 45W 11.25W 2x DDR3-1333 $313(check) $78.25 $39.13 3840 12.27 0.27 1265LV2 Riseup has 5+
Model Family Socket Clock Turbo Cores (Threads) Cache Cache/ Core TDP TDP/ Core RAM Cost Cost/ Core Cost/ MB CPUMark CPUmark/$ CPUmark/$/W Replacement Notes

Desktop CPUs

These are “Desktop” only in the sense that they are marketed for Desktop and a cheaper price (at the cost of cache usually). In most cases it makes more sense to get the above “server” models, but if these are good enough and price is an issue, maybe these are interesting?

Model Family Socket Clock Turbo Cores (Threads) Cache Cache/ Core TDP TDP/ Core RAM Cost Cost/ Core Cost/ MB CPUMark CPUmark/$ CPUmark/$/W Replacement Notes
i7-4770T Haswell LGA1150 2.0GHz 3.0GHz 4(8) 9MB 2.25MB 45W 11.25W 2x DDR3-1600 $303?(check) $? $? ?
i7-4765T 2.0GHz 3.0GHz 4(8) 9MB 2.25MB 35W 11.25W $303?(check) $? $? ?
i5-4660T 2.3GHz 3.3GHz 4(4) 6.5MB 1.125MB 45W 11.25W $213?(check) $? $? ?
i5-4570T 2.9GHz 3.6GHz 2(4) 4.5MB 2.25MB 35W 17.5W $192?(check) $? $? ?
i7-3770T Ivy Bridge LGA1155 2.5GHz 3.7GHz 4(8) 8MB 2MB 45W 11.25W 2x DDR3-1600 $295(check) $73.75 $36.86 8654 29.44 0.65
i5-3570T 2.3GHz 3.3GHz 4(4) 6MB 1.5MB 45W 11.25W $206(check) $51.50 $34.33 6155 29.88 0.66
i5-3470T 2.9GHz 3.6GHz 2(4) 3MB 1.5MB 35W 17.5W $?(check) $ $
i3-3240T 2.9GHz N/A 2(4) 3MB 1.5MB 35W 17.5W $206(check) $? $? -
i3-3220T 2.8GHz N/A 2(4) 3MB 1.5MB 35W 17.5W $130(check) $65 $43.33 3813 30.51 0.87
G2100T 2.6GHz N/A 2(2) 3MB 1.5MB 35W 17.5W $?(check) $ $
G2020T 2.5GHz N/A 2(2) 3MB 1.5MB 35W 17.5W 2x DDR3-1333 $(check) $ $
G1610T 2.3GHz N/A 2(2) 2MB 1MB 35W 17.5W $(check) $ $

Mainboards

We like the supermicro boards with IPMI.

AMD SP3

Epyc2 systems

Servers

Description Model Price Notes
H12 WIO 2.5" bays 1114S-WTRT $1255 1U, 10 2.5", WIO
H12 WIO 3.5" bays 1014S-WTRT $1165 1U, 4 3.5", WIO
H12 Big Twin server 2124BT-HTR $4630 2U, 4 nodes, 24 2.5"
H12 TwinPro 2014TP-HTR $? 2U, 4 nodes, 12 3.5"

Mainboards

Description Model Price Notes
H12 WIO single H12SSW-NT $455 WIO
H12 BigTwin dual H12DST-B N/A Dual Socket
H12 TwinPro single H12SST-PS N/A Single Socket

LGA1151:

All Supermicro LGA1151 single socket boards.

Model Form SATA Price Notes
X11SSH-F microATX 8 SATA3 $207 1xM.2
X11SSM-F $196
X11SSL-F 6 SATA3 $174
X11SSL-nF $193 2xNVMe
X11SSW-F WIO $213 1xM.2

LGA2011v3:

All Supermicro LGA2011v3 single socket boards.
All Supermicro LGA2011v3 dual socket boards.

Model Sockets Form DIMMs SATA LAN Price Notes Cases
X10SRL-F 1 ATX 8 10 SATA 6.0g 2×1g $263.50
X10SRi-F E-ATX 16 $276.50
X10DRL-i 2 ATX 8 $319.25
X10DRi E-ATX 16 $396.50
X10DRi-LN4+ EE-ATX 24 4×1g $517.50

LGA1150:

All Supermicro LGA1150 boards.
Boards with IPMI

Model Form DIMMs SATA LAN Price Notes Cases
X10SL7-F uATX 4 dimms 2xSATA 6gb, 4xSATA 3gb 2xGb lan $202.95 Additional SAS controller
X10SLA-F ATX 4 dimms 2xSATA 6gb, 4xSATA 3gb 2xGb lan $185.15
X10SLH-F uATX 4 dimms 6xSATA 6gb 2xGb lan $201.85
X10SLL-F uATX 4 dimms 2xSATA 6gb, 4xSATA 3gb 2xGb lan $154.95
X10SLL-SF uATX 2 dimms 2xSATA 6gb, 2xSATA 3gb 2xGb lan
X10SLM-F uATX 4 dimms 4xSATA 6gb, 2xSATA 3gb 2xGb lan $167.50 2 different Gb lan chipsets
X10SLM+-F uATX 4 dimms 4xSATA 6gb, 2xSATA 3gb 2xGb lan $191.75 2x Identical Gb lan (better than non-plus version)
X10SLM+-LN4F uATX 4 dimms 4xSATA 6gb, 2xSATA 3gb 4xGb lan $205.25 4x Identical Gb lan

Various vendor Mini-ITX boards

Model Details Price Notes
ASUS H87I Plus hardware.info page $114
ASUS Z87-ITX $? Not released yet
MSI Z87I hardware.info page $?
ASRock Z87E-ITX hardware.info page $165
ASRock H87M-ITX hardware.info page $104
ASRock B85M-ITX hardware.info page $80
Gigabyte Z87N WiFi hardware.info page $138

LGA1155:

Boards with IPMI

Model Price Notes Cases
X9SCM-F $171.65@interpro MicroATX, 2GbE ports(different), 2 SATA 6Gbps, 4 SATA 3Gbps SC510T-200B
X9SCL+-F $179.50@interpro MicroATX, 2GbE ports(identical), 6 SATA 3Gbps SC510T-200B
X9SCL-F $161.35@interpro MicroATX, 2GbE ports(different), 6 SATA 3Gbps SC510T-200B
X9SCA-F $191.55@interpro ATX, 2GbE ports, 2 SATA 6Gbps, 4 SATA 3Gbps
X9SCi-LN4F $202.50@interpro ATX, 4GbE ports, 2 SATA 6Gbps, 4 SATA 3Gbps
X9SPU-F $204.95 UIO, 2GbE ports, 2 SATA 6Gbps, 4 SATA 3Gbps SC111LT-330U

All Supermicro LGA1155 boards

LGA2011:

Model Form Factor Max RAM RAM Slot SATA ports Network, Features Price Notes
X9DRi-F E-ATX 512GB DDR3 16 DIMM slots 2 SATA3, 8 SATA2 Intel i350 GbE $423.75 @interpro(check)
X9DRD-EF E-ATX 512GB DDR3 16 DIMM slots 2 SATA3, 4 SATA2 Intel i350 GbE ?(check) Interpro doesn’t have it, only 6 sata ports which makes it less interesting for the SC113/116 cases
X9DRH-iF E-ATX 512GB DDR3 16 DIMM slots 2 SATA3, 8 SATA2 Intel i350 GbE $459.45 @interpro(check)
X9DRW-iF E-ATX, WIO 512GB DDR3 16 DIMM slots 2 SATA3, 8 SATA2 Intel i350 GbE $437.25 @interpro(check) The only SC113 case with WIO is 700W
X9DRX+-F Proprietary 512GB DDR3 16 DIMM slots 2 SATA3, 8 SATA2 Intel i350 GbE, 11 PCI-E slots $491.50 @interpro(check)
X9DRG-QF Proprietary 512GB DDR3 16 DIMM slots 2 SATA3, 8 SATA2 Intel i350 GbE, 4 x16 PCI-E non-blocking $497.50 @interpro(check)

All Supermicro LGA2011 boards

Starting with the full list, to get the above table we do:

  • no need for 4 gig ports => drop the LN4 ones
  • need IPMI: drop those without “F” in the name
  • no need for 10G => drop the those with “T” in the postfix of the name
  • need >= 16 DIMM slots for interleaving
  • no need for SAS

LGA1156:

We only used one, the Supermicro X8SIL-F, $171.50@interpro. It works with both the SC113MTQ-330B and SC510T-200B cases.

All Supermicro LGA1156 boards

Chassis

low power, not many disks

Model PS Wattage Drive bays Price Notes
SC813MT-300C 300W 4×3.5″ $230
SC813MTQ-350CB 350W 4×3.5″ $280
SC811T-260 260W 2×3.5″ $175
SC813MTQ-280C 280W 4×3.5″ $250 shorter than the TQ version, smaller mainboard spot but still big enough for boards we want. Interpro sells a version of this with a 330W PS for $273.50
SC113MTQ-330B 330W 8×2.5″ $273.50
SC502-200B 200W 2×2.5″ fixed $100 might need active heatsink
SC503-200 200W 2×2.5″ fixed $90 front i/o, might need active heatsink
SC502L-200B 200W 2×2.5″ fixed $90 for Atom, no fans
SC503L-200B 200W 2×2.5″ fixed $76 for Atom, no fans
SC510T-200B 200W 2×2.5″ hotplug $142.50
SC510T-203B 200W(80 gold) 2×2.5″ hotplug $?
Casetronic C159-60W 60W 1×3.5″ fixed $240
KI-N145 300W 2×3.5″ fixed ~$90 $28 for case, $35 for 300W PSU, $12 for rails, $2 for Riser card (right hand), $10-20 for CPU cooler
KI-M550 2×200W 4×3.5″(or 8×2.5″) fixed ~$106 $50 for case, $44 for 2 200W PSU, $12 for rails. Can’t support PCI cards or larger than mini-ITX.

Some things to consider about short chassis

  • Short cases don’t have rails, so you’d have to unscrew/rescrew them each time you needed to access them, or come up with some sort of short rails to slide them in on.
  • Maybe it’s not as important to have hotplug for SSDs, hopefully they don’t fail as often (but if they are fixed, that compounds the above problem
  • Rear I/O on short cases might be a pain as the cabling would need to be way forward in the rack
  • Front I/O would let us mount cases on both sides, although we’d need a way to reach the power cord. Supermicro doesn’t seem to have a Front I/O case with power on the front too. (but we could use a picoPSU)
  • In order to get a hotplug disk on the I/O side of a machine, we could use one of the PCI slot adapters listed below.
  • I asked supermicro and the cases aren’t designed to move the mounting brackets to the other side in order to turn a Rear I/O into a Front I/O style machine. We might still be able to do that if we weren’t relying on those tabs for holding the machine up.
  • I mailed Supermicro to ask about the 503* compatibility with the Atom board and X8SIL (Ibex Peak) board that we are interested in using, I’ll update when they reply.

We’re not super happy with any of the above, here is the ultimate case we’d build if we could.

lots of disks

(last updated 2012-05-28)

Model Height Drive Size PS Wattage Drive bays Drive/U Price Similar models/Notes
SC826TQ-R500LPB 2U 3.5" 500W 12 6 $665(check) E16
SC936A-R900B 3U 900W 16 5.3 $880(check) E1 for $1075
SC837E16-RJBOD1 1620W 28 9.3 $1502(check) E26
SC846TQ-R900B 4U 900W 24 6 $990(check) SC846A-R900 $1075, E1-R900 $1125, E2-R900 $1275
SC116TQ-R700CB 1U 2.5" 700W 10 10 $550(check) UIO $670, WIO, $675
SC213A-R740LPB 2U 740W 16 8 $705(check) WIO $705, SC213A-R720LPB (720W) $700, they make 8 bay versions but it would be better to use a 1U
SC216A-R900LPB 900W 24 12 $863(check) UIO $870, E1+UIO $1035, E2 $1240, E2+UIO $1250
SC417E16-RJBOD1 4U 1400W 88 22 $2550(check)
SC417E16-R1400LPB 72 18 $2180(check) UIO $2190, E26 $2705, E26+UIO $2715

Normal (in 2U+) = 7 Low-profile
UIO = Universal I/O, vertical I/O that lets you install 4x Full-height expansion slots and 3x Low-Profile Expansion Slots
WIO = 4x Full-height, Full-length and 3x Low-profile expansion slots
E1 = single SAS2 3Gb expander
E2 = dual SAS2 3Gb expander
E16 = single SAS2 6Gb expander
E26 = dual SAS2 6Gb expander

Desktop Chassis

Mini-ITX

Model Price Drives PSU Fans
CHENBRO SR30169T2-250 $140 4×3.5″ hotswap, 1×2.5″ ATX 250W 1 120mm fan
IN WIN BP655.300TB3L $65 1 5.25", 1 3.5" TFX 200W 1 80mm
LIAN LI PC-Q08B $110 1 5.25", 6 3.5" ATX, none 1 120mm, 1 140mm
LIAN LI PC-Q25B $120 5 3.5" hot swap, 2 2.5" ATX, none 1 120mm, 1 140mm
SilverStone Sugo Series SG05BB-LITE $40 1 3.5", 1 2.5" SFX, none 1 120mm
SilverStone Sugo Series SG05B-USB3.0 $107 1 3.5", 1 2.5" SFX, 300W 80plus 1 120mm

SATA Controllers

Model PCI-X Connector Ports Price Price/Port
LSI 9211-4i x4 1 x4 SFF8087 4 $183.50 $45.88 RAID
LSI 9211-8i x8 2 x4 SFF8087 8 $249.65 $31.21 RAID
LSI 9240i-4i x8 1 x4 SFF8087 4 $207.35 $51.84 RAID
LSI 9240-8i x8 2 x4 SFF8087 8 $279.65 $34.96 RAID
Supermicro AOC-SAS2LP-MV8 x8 2 x4 SFF8087 8 $107.50 $13.44 No RAID

HDD Disks

(this section deleted and redone Jan 2015)

Notes

  • backblaze.com is cloud storage company that designs and builds their own chassies that hold huge amounts of inexpensive drives. They have a blog where they document all sort of interesting things: HDD failure rates, experience with 6tb drives, what SMART stats they monitor. Keep in mind their workload is bulk storage, they are using doing large block streaming reads and writes. But their data about drive failure rates is still interesting.
  • Anandtech.com has an HDD category with good reviews.
  • This page in anandtech’s 4tb review has good descriptions of the various vendor product lines.

3.5" SATA HDDs

Product lines:

Product Line Sizes RPMs Cache Notes
WD Black 4/3/2/1/0.5tb 7200 64 consumer performance (compared to Blue)
WD Re 4/3/2/1/0.5/0.25tb 7200 64 datacenter performance
WD Xe 900/600/450/300gb 10000 32 datacenter high performance (SAS only)
WD Se 4/3/2/1tb 7200 64 datacenter capacity
WD Ae 6.3tb 5760 64 datacenter archive
WD Red 6/5/4/3/2/1tb IntelliPower(slow) 64mb consumer NAS, 1-8 bay, 3y warranty
WD Red Pro 4/3/2tb 7200 64 consumer performance NAS, 8-16 bay, 5y warranty
HGST MegaScale DC 4000.B 4tb CoolSpin(slow) 64 low-workload, low-power
HGST Ultrastar 7K4000 4/3/2tb 7200 64 datacenter performance (previous gen)
HGST Ultrastar 7K6000 6/5/4/2tb 7200 128 datacenter performance
HGST Deskstar Kits 4/3/2 7200 or CoolSpin 32 or 64 desktop
HGST Deskstar NAS Kits 6/5/4/3tb 7200 64 or 128 NAS, anandtech review

FIXME: need Seagate (warning, some of the Seagates suck, see the backblaze report), maybe Toshiba

WD Red Pro:

Model Size Price Price/tb
WD2001FFSX 2tb $145 $72.50
WD3001FFSX 3tb $170 $56.67
WD4001FFSX 4tb $227 $56.75

WD Red:

Model Size Price Price/tb
WD10EFRX 1tb $64 $64.00
WD20EFRX 2tb $96 $48.00
WD30EFRX 3tb $114 $38.00
WD40EFRX 4tb $185 $46.25
WD50EFRX 5tb $214 $42.80
WD60EFRX 6tb $270 $45.00

HGST Deskstar NAS:

Model Size Price Price/tb
0S03660 3tb $125 $41.67
0S03664 4tb $185 $46.25
0S03835 5tb $267 $53.40
0S03839 6tb $290 $48.33

FIXME: do price tables for other lines

SSD Disks

(These notes are old)

They are getting affordable for the small ones. Blazing fast, might be good for certain workloads. You can get decent 30-40gb drives for $100-120 now. You can get cheaper in other brands, but you have to be careful which controllers they have, some really suck.

Here’s what I have determined from reading a ton of articles:

  • Of all the early SSD drives, most lost performance over time. Some you could get back to new performance by using a reset utility that told the drive to start over. Intel solved this problem first with their 2nd generation drives.
  • Early SSD drives had limitations on the number of write cycles the flash could do, such that you might actually run into it. Modern controllers/firmware and improved flash in newer generation drives mean you won’t realistically ever run into this problem.
  • There is an ATA command called TRIM that tells the device when a block is considered no longer used by the O/S and allows the drive to recycle it, thus helping prevent the lost performance over time problem. Support for this in Linux doesn’t show up until 2.6.33. I don’t know if it works with MD devices, probably not? For now we shouldn’t count on needing TRIM support for a drive to work well, better to pick drives that don’t have the problem.
  • Intel made the best SSDs for the first couple of generations and their stuff is still solid although currently not the performance leader and not the cheapest per GB. It’s been a while since their G2 stuff was released and while it was the fastest when it came out, it’s not anymore.
  • All drives of a particular SSD generation&brand use the same controller and NAND chips, and the controller is designed to support a certain number of chips, which is the max size drive that generation/brand support. They make smaller sizes by not fully populating the drive with chips. The NAND chips are accessed in parallel, so the largest drive is the fastest and it drops off as you get smaller. However even the smallest size is way faster than HDDs.
  • The Indilinx controller mostly sucks, we should avoid it.
  • SandForce is a small company that makes a controller used by several vendors including OCZ and Corsair. The current controllers are pretty awesome and the next generation is insanely awesome. Here’s a couple articles.
  • One of the reasons Sandforce is so much faster in benchmarks is that they use this “durawrite™” algorithm to reduce the writes they need to do to NAND by doing compression and deduplication of data. But if your data is more random (like with O/S based encryption) then it loses the benefit. This new AnandTech review (of a drive that’s not that interesting) includes graphs and data of many drives being tested with random data. Here’s another review that also mentions the random data issue. His conclusion is that a SF-1200 with random data is slower than the other Intel/Indilinx offerings available (but he’s talking higher end). It’s unclear how they would compare to lower end stuff like the Intel X25-V.
  • Intel’s 3rd generation using 25nm flash is expected in Q4 2010 – Q1 2011 and will be quite a bit better than their G2, but only on par with Sandforce’s current stuff and slower than Sandforce’s next gen stuff.
  • This article about Crucial’s RealSSD C300 product, which uses a Marvell controller, says that it’s fast but that it has a problem with performance degrading after the drive has been used a lot. We should avoid it.
  • Intel dropped prices on the X25-M G2 line on Nov 11, 2010 and also introduced a drive in the middle of the line. Unclear how/when this will affect the X25-V drive.
  • Of the SandForce partners, OCZ has an exclusive deal for the Vertex 2 drive where it’s firmware is better at small random writes, the Agility 2 doesn’t have this and also writes are capped at 50mb/s. Corsair somehow is shipping this faster firmware (3.0.1), but future versions will likely add this crippling.

SSD Models

(This chart is pretty old now, see the Samsung 850 EVO stuff below)

Model Bus Controller Flash Prices Info
Intel 320 SATAII Intel MLC 25nm 40gb $90(newegg) / 80gb $150(newegg) / 120gb $180(newegg)
40gb $96(interpro) / 80gb $174(interpro) / 120gb $204(interpro)		 anantech riseup has a lot of these deployed
Intel 330 SATAIII Sandforce SF-2281 MLC 25nm 180gb $175(newegg) / 240gb $210(newegg)
60gb $84(interpro) / 120gb $134(interpro) / 180gb $190(interpro)		 anandtech. the 330 and the 520 are identical except the 520 uses better flash, so we should prefer that when the price is the same
Intel 335 SATAIII Sandforce SF-2281 MLC 20nm 180gb $175(newegg) / 240gb $210(newegg)
180gb $188(interpro) / 240gb $204(interpro)		 anandtech. the anandtech review leads me to believe the 335 is worse than the 520, so unless it’s a really good deal we should prefer the 520
Intel 520 SATAIII Sandforce SF-2281 MLC 25nm 60gb $85(newegg) / 120gb $150(newegg) / 180gb $195(newegg)
60gb $107(interpro) / 120gb $147(interpro) / 180gb $214(interpro)		 anandtech. the 330 and the 520 are identical except the 520 uses better flash, so we should prefer that when the price is the same
Intel 530 SATAIII ? ? 80/120/180/240/360/480
Intel DC3700 SATAIII Intel MLC-HET 25nm expensive anandtech DC=data center higher, quality NAND (overkill for us)
Samsung 830 SATAIII Samsung MCX MLC 64gb $90(newegg) / 128gb $115(newegg) / 256gb $160(newegg) anandtech
Samsung 840 SATAIII Samsung MDX TLC 120gb $95(newegg) / 250gb $210(newegg) anandtech
Samsung 840 Pro SATAIII Samsung MDX MLC 64gb $100(newegg) / 128gb $140(newegg) / 256gb $250(newegg) anandtech
Samsung 840 EVO SATAIII Samsung MEX TLC 120gb $110/250gb $190/500gb $370/750gb $530/1tb $650 anandtech, the 250gb and 750gb are the most interesting for cache reasons
Intel 313 mSATA ? ? 20gb $100(newegg)
Intel 525 mSATA Sandforce SF-2281 MLC 25nm 30gb $60(newegg) anandtech

Samsung 850 EVO

Samsung 850 EVO (anandtech review) are currently (2014-12) the most interesting. Prices, updated 2014-12-31

Size Interpro Newegg Amazon Cheapest/gb
1tb N/A $469.99 $464.99 0.465
500gb N/A $253.99 $249.00 0.498
250gb N/A $139.99 $139.00 0.556
120gb N/A $ 94.99 $ 89.00 0.742

General info

comparison of Intel 520/330/335
Information about SSD "spare area" – the cool thing is that you can adjust spare area by just not using part of the drive. This could be done by making the partition table smaller than the whole drive. This would increase write performance, make performance more consistent, decrease write amplification, decrease writes which would increase the life of the flash.
Flash endurance, and another

m.2 drives

m.2 is a mini card PCI Express form factor. The cards come in various lengths and it’s being used for wireless cards, SSDs, and a bunch of other things. Some newer systems have built-in m.2 slots, but you can also buy PCIe x4 adapter cards (which are pretty simple, just rerouting the PCIe lines, no logic on the card).

The most interesting m.2 devices are SSDs that speak PCIe x4 which is a big increase in bandwidth over SATA 6g. The Samsung XP941 m.2 SSDs are marketed as being capable of:

Capacity 128GB 256GB 512GB
Sequential Read 1000MB/s 1080MB/s 1170MB/s
Sequential Write 450MB/s 800MB/s 950MB/s
4KB Random Read 110K IOPS 120K IOPS 122K IOPS
4KB Random Write 40K IOPS 60K IOPS 72K IOPS

Here is a good anandtech article with m.2 background and review of the Samsung XP941.

Adapters

Model PCIe Sizes Supported Price Notes
ADDONICS ADM2PX4 3.0 x4 42, 60, 80 $27.50
M Factors M2P4A 2.0 X4 42, 60, 80, 110 $59 heatsink, is only PCIe 2.0?
M Factors DT-120 3.0 x4 42,60,80 $24 aka Lycom
M Factors M2P4S ?.0 x4 42, 60, 80, 110 $50 two m.2 drives using flex cable($62)

Drives

Model PCIe length sizes prices notes
Samsung XP941 2.0 x4 80mm 128, 256, 512 $143/$260/$510 anandtech
Samsung SM951 3.0 x4 80mm 128, 256, 512 N/A anandtech, MLC
Samsung PM951 3.0 x4 80mm 128, 256, 512 N/A TLC
Intel DC S3500 3.0 x4 80mm 80, 120, 160, 340 N/A

Drive Adapters

install two 2.5" drives in a single 3.5" drive spot.

install 2.5" drives in a PCI slot

install four+ 2.5" drives in a 5.25" case bay

install two 2.5" drives plus a slim optical drive in a 5.25" case bay

install two 2.5" drives in a 3.5" case bay

install a single 2.5" SATA drive in a 3.5" hotplug tray

RAM

LRDIMM

There exists a new thing called Load Reduced DIMMs (LRDIMMs). They are for putting tons of RAM in the system (but at a slight latency cost). Currently 32gb and 64gb LRDIMMs are available, allowing a system with 16 slots to have 512gb or 1tb. More about them: 1 2 3

Crucial.com RAM prices

prices last updated: 2010-02-15

Generation Interface Type Registered ECC 4gb 2gb 1gb
DDR3 SODIMM DDR3-1333 aka PC3-10600 NO NO $250 $60 $32
DDR3-1066 aka PC3-8500 NO NO $250 $60 $32
DIMM DDR3-1333 aka PC3-10600 NO NO $250 $60 $32
DDR3-1333 aka PC3-10600 NO YES $230 $75 $45
DDR3-1333R aka PC3-10600R YES YES $184 $95 $56
DDR3-1066 aka PC3-8500 NO NO $??? $64 $32
DDR3-1066 aka PC3-8500 NO YES $230 $75 $45
DDR3-1066R aka PC3-8500R YES YES $184 $95 $56
DDR2 DIMM DDR2-1066 aka PC2-8500 NO NO $??? $66 $35
DDR2-800 aka PC2-6400 NO NO $??? $48 $26
DDR2-800 aka PC2-6400 NO YES $??? $63 $32
DDR2-800R aka PC2-6400R YES YES $170 $80 $55
DDR2-667 aka PC2-5300 NO NO $190 $48 $26
DDR2-667 aka PC2-5300 NO YES $200 $63 $32
DDR2-667R aka PC2-5300R YES YES $161 $90 ?

Laptops

Laptop as primary computer

Some people choose to use a laptop as their primary computer. For them, durability, screen size, and performance are much more important than cost.

FIXME: Someone other than taggart will need to write this.

On a sort of related note, dkg’s TrustedPhysicalConsole might be relevant here (and also below, but more so here).

Laptop as cafe network terminal

If you primarily use a powerful desktop at home and just need something to let you do some basic things while out at a cafe or on trips, portability (size and weight), “good enough” performance, and battery life are more important than durability, screen size, high-performance. It’s probably not as tricked out as the above, isn’t used for all tasks, probably doesn’t hold crypto keys, and is mainly a way to securely access “real” computers. It probably doesn’t hold any unique data for long periods of time, just checked out git trees that are also elsewhere, etc.

I (taggart) have been using an Asus eeepc (10" screen, Atom CPU, 2g RAM, 6h battery) since early 2009 and it’s still more than fast enough to run a bunch of terminals, a web browser with a couple dozen tabs, play music and internet videos, run git, etc. However, there are some similar but much newer things available. Here are some notes I’m taking to help decide.

model CPU Cores Freq RAM Disk Display Resolution Weight Dimensions Interfaces Price Notes
Asus eeebook x205 BayTrail-T Z3735 4(4) 1.33 2gb 32 or 64gb SSD 11.5" 1366×768 2.2lbs 11.3"W x 7.6"D x 0.7"H microHDMI, microSD, 2x USB2, SD camera $199(32gb) product page, CNET review,NetworkWorld review, may be hard to install BIOS doesn’t let you boot, comes in Red and Black!
Asus Zenbook UX305 CoreM 5Y10/5Y71 2(4) 0.8(2.0) 4 or 8gb 128/256/512gb SATA3 SSD 13.3" 1920×1080 or 3200×1800 2.76lbs 12.0"W x 8.9"D x 0.5"H microHDMI, SD, 3x USB3, HD camera $? product page, anandtech review
Toshiba Chromebook 2 Celeron N2840 2(2) 2.16 2gb , 4gb 16gb 13.3" 1366×786, 1920×1080 3lb 12.60 × 8.40 × 0.76 HDMI, SD, 1x USB3, 1x USB2 HD camera $230(2G,SD), $330(4G,HD) product page,wired review, cnet review
   
Micah
2010-10-12
I emailed supermicro to ask if these were compatible (you have to check, because 1U servers can’t use a ATX style I/O port shield, it’s too tall for 1U) and they said they were

this worries me a little because they told you that before and it was wrong :(