Hi Ben, Can we have an access to one of the teradimm block devices? or can we move some of exFlash DIMMs to other system? John is interested in trying out teradimm. I'm also interested in hacking the kernel module. BTW, do you know the cost of each DIMM? do we have any other NVRAM devices in JLSE? - kaz -------- Forwarded Message -------- Subject: [JLSE-Discuss] IBM x3650 M4 with exFlash SSD DIMMs (Diablo Technologies Teradimm) - wally.ftm.alcf.anl.gov Date: Fri, 8 Aug 2014 16:52:48 +0000 From: Allen, Benjamin S. <[email protected]> To: [email protected] <[email protected]> JLSE has installed a IBM x3650 with 8 400GB exFlash modules in JLSE as "wally". It only has a 1GbE connection currently. Initial benchmarks of the DIMMs are pretty good. Currently Kevin Harms is working on the machine. The devices are presented as block devices /dev/td{a..h}. The cli tool is "td-cli", and the kernel module is "teradimm". teradimm_ipmi_proxy communicates the status of the SSD DIMMs to the IMM/BMC. The DRAM and SSD DIMMs are all running at 1333Mhz for some reason even though all the modules installed are capable of 1600Mhz (1866Mhz for the DRAM). By IBM's published stats, this lowers throughput but actually increases IOPs performance a bit. I've found no way to force them into 1600Mhz in the BIOS. The SSD DIMMs have been updated with the latest firmware from IBM. I've also made sure the uEFI, IMM, and pDSA firmware versions are at least at the minimum levels to support exFlash DIMMs. I've also ensured the BIOS settings mentioned on http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=SERV-FLASHDM are all set correctly. It appears the devices are optimized for 4KB IO, throughput is even good at 4K, albeit a bit better at 64KB. It also appears they do better with only a single thread per DIMM. However this is based on about 30 minutes of benchmark with btest at this point. With a single thread at 4K latency is measuring at 8 usec, which is pretty insane for SSD. [bsallen@wally IBM]$ sudo /tmp/btest -b 4096 -T 2 S R /dev/tda /dev/tdb /dev/tdc /dev/tdd /dev/tde /dev/tdf /dev/tdg /dev/tdh IO Block size is 4096 Use sync IO mode Using the default 60 seconds time limit Use pure sequential IO Generate only reads 16 threads are ready, starting test at Thu Jul 31 22:19:58 2014 Subtotal (diff): 16 workers, 9.832 seconds (9.818), 741134.500 iops, avg latency 21 usec, bandwidth 2964528 KB/s, errors 00, ops 7287111 Subtotal (diff): 16 workers, 9.832 seconds (9.818), 1171 max_latency, hiccups levels: <1ms: 7287103 1ms: 8 2-10ms: 0 11-50ms: 0 51-100ms: 0 >100ms: 0 Subtotal (diff): 16 workers, 9.737 seconds (19.547), 734156.938 iops, avg latency 21 usec, bandwidth 2936624 KB/s, errors 00, ops 7148447 Subtotal (diff): 16 workers, 9.737 seconds (19.547), 1093 max_latency, hiccups levels: <1ms: 7148441 1ms: 6 2-10ms: 0 11-50ms: 0 51-100ms: 0 >100ms: 0 Subtotal (diff): 16 workers, 9.735 seconds (29.274), 737060.625 iops, avg latency 21 usec, bandwidth 2948240 KB/s, errors 00, ops 7175456 Subtotal (diff): 16 workers, 9.735 seconds (29.274), 1021 max_latency, hiccups levels: <1ms: 7175453 1ms: 3 2-10ms: 0 11-50ms: 0 51-100ms: 0 >100ms: 0 Subtotal (diff): 16 workers, 9.735 seconds (39.001), 736089.625 iops, avg latency 21 usec, bandwidth 2944352 KB/s, errors 00, ops 7166093 Subtotal (diff): 16 workers, 9.735 seconds (39.001), 7815 max_latency, hiccups levels: <1ms: 7166084 1ms: 5 2-10ms: 4 11-50ms: 0 51-100ms: 0 >100ms: 0 Subtotal (diff): 16 workers, 9.756 seconds (48.740), 693506.188 iops, avg latency 23 usec, bandwidth 2774016 KB/s, errors 00, ops 6765596 Subtotal (diff): 16 workers, 9.756 seconds (48.740), 8052 max_latency, hiccups levels: <1ms: 6765589 1ms: 3 2-10ms: 4 11-50ms: 0 51-100ms: 0 >100ms: 0 Total: 58.554 seconds, 722776.438 iops, avg latency 22 usec, bandwidth 2891104 KB/s, errors 0, total ops 42321669 Total: 58.554 seconds, 8052 max_latency, hiccups levels: <1ms: 42321608 1ms: 43 2-10ms: 18 11-50ms: 0 51-100ms: 0 >100ms: 0 Test is done at Thu Jul 31 22:20:59 2014 [bsallen@wally IBM]$ sudo /tmp/btest -b 4096 -T 2 S W /dev/tda /dev/tdb /dev/tdc /dev/tdd /dev/tde /dev/tdf /dev/tdg /dev/tdh IO Block size is 4096 Use sync IO mode Using the default 60 seconds time limit Use pure sequential IO Generate only writes 16 threads are ready, starting test at Thu Jul 31 22:22:03 2014 Subtotal (diff): 16 workers, 9.906 seconds (9.901), 554011.438 iops, avg latency 28 usec, bandwidth 2216032 KB/s, errors 00, ops 5488264 Subtotal (diff): 16 workers, 9.906 seconds (9.901), 482 max_latency, hiccups levels: <1ms: 5488264 1ms: 0 2-10ms: 0 11-50ms: 0 51-100ms: 0 >100ms: 0 Subtotal (diff): 16 workers, 9.814 seconds (19.711), 546904.312 iops, avg latency 29 usec, bandwidth 2187616 KB/s, errors 00, ops 5367422 Subtotal (diff): 16 workers, 9.814 seconds (19.711), 1772 max_latency, hiccups levels: <1ms: 5367420 1ms: 2 2-10ms: 0 11-50ms: 0 51-100ms: 0 >100ms: 0 Subtotal (diff): 16 workers, 9.812 seconds (29.525), 553523.938 iops, avg latency 28 usec, bandwidth 2214080 KB/s, errors 00, ops 5431129 Subtotal (diff): 16 workers, 9.812 seconds (29.525), 1032 max_latency, hiccups levels: <1ms: 5431127 1ms: 2 2-10ms: 0 11-50ms: 0 51-100ms: 0 >100ms: 0 Subtotal (diff): 16 workers, 9.811 seconds (39.331), 553460.688 iops, avg latency 28 usec, bandwidth 2213840 KB/s, errors 00, ops 5430267 Subtotal (diff): 16 workers, 9.811 seconds (39.331), 549 max_latency, hiccups levels: <1ms: 5430267 1ms: 0 2-10ms: 0 11-50ms: 0 51-100ms: 0 >100ms: 0 Subtotal (diff): 16 workers, 9.812 seconds (49.139), 553227.312 iops, avg latency 28 usec, bandwidth 2212896 KB/s, errors 00, ops 5428196 Subtotal (diff): 16 workers, 9.812 seconds (49.139), 544 max_latency, hiccups levels: <1ms: 5428196 1ms: 0 2-10ms: 0 11-50ms: 0 51-100ms: 0 >100ms: 0 Total: 58.970 seconds, 551339.438 iops, avg latency 29 usec, bandwidth 2205344 KB/s, errors 0, total ops 32512339 Total: 58.970 seconds, 1804 max_latency, hiccups levels: <1ms: 32512333 1ms: 6 2-10ms: 0 11-50ms: 0 51-100ms: 0 >100ms: 0 Test is done at Thu Jul 31 22:23:04 2014 References: http://www.diablo-technologies.com/ http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=SERV-FLASHDM exFlash DIMM v1.4.7 software including kernel module source: http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5095918 _______________________________________________ Discuss mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/discuss