March 2015 - Admins - mailman3.cels.anl.gov

Re: [JLSE-Admins] [JLSE-Discuss] IBM x3650 M4 with exFlash SSD DIMMs (Diablo Technologies Teradimm) - wally.ftm.alcf.anl.gov
by Kazutomo Yoshii 17 Mar '15

17 Mar '15

Hi Ben, Thank you for sudo access! John, Let's fill the request form together if exFlash SSD still sounds good for you. I guess other CESAR folks are interested in this device. If so, we can add this to a request. Here is a list of the current JLSE projects. http://jlse.anl.gov/projects/ - kaz On 03/17/2015 01:04 PM, Allen, Benjamin S. wrote: > Hi Kaz, > > Sudo access is fine. I'll add it later today. > > Would you or John please fill out a project request: http://press3.mcs.anl.gov/jlse/projects/project-request-form/, so we can list the project description? > > We don't have a start date for the student yet. As far as exclusivity it will depend on how quick the student gets spun up, but more or less I would assume yes. > > Ben > >> On Mar 16, 2015, at 1:32 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >> >> Ben, >> >> Thanks! >> >> John and I would like to have a sudo access, so that >> we can try something before the student starts working here. >> >> Do you know when the student starts? >> I assume that the student use this machine exclusively when he starts here. >> >> - kaz >> >> On 03/16/2015 01:25 PM, Allen, Benjamin S. wrote: >>>> On Mar 16, 2015, at 1:15 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >>>> >>>> Hi Ben, >>>> >>>> Can we have an access to one of the teradimm block devices? >>>> or can we move some of exFlash DIMMs to other system? >>> We only have one system. We have a summer student this year under Kevin Harms thats slated todo research on the box, so it can be used till then. I'll need to be able to put back to normal before the student starts. >>> >>> Hostname is "wally.ftm.alcf.anl.gov". Let me know if you need sudo access. >>> >>>> John is interested in trying out teradimm. >>>> I'm also interested in hacking the kernel module. >>> Feel free to mess with the kernel module. No editing the BIOS settings on the box however. The BIOS appears to be vary fragile. It's very easy to get the machine into a state that doesn't boot. >>> >>>> BTW, do you know the cost of each DIMM? >>>> do we have any other NVRAM devices in JLSE? >>> We ordered the box last April. As a whole the server costs $29,611.00 with 8 exFlash DIMMs installed. The quote shows the list price as $4,949.00 for the 400GB exFlash DIMM. Our overall discounted price for the server was 44% off list price. >>> >>> No we don't have any other. I don't know of any other nvram product actually shipping yet. >>> >>> Ben

1 0

Re: [JLSE-Admins] [JLSE-Discuss] IBM x3650 M4 with exFlash SSD DIMMs (Diablo Technologies Teradimm) - wally.ftm.alcf.anl.gov
by Allen, Benjamin S. 17 Mar '15

17 Mar '15

Hi Kaz, Sudo access is fine. I'll add it later today. Would you or John please fill out a project request: http://press3.mcs.anl.gov/jlse/projects/project-request-form/, so we can list the project description? We don't have a start date for the student yet. As far as exclusivity it will depend on how quick the student gets spun up, but more or less I would assume yes. Ben > On Mar 16, 2015, at 1:32 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: > > Ben, > > Thanks! > > John and I would like to have a sudo access, so that > we can try something before the student starts working here. > > Do you know when the student starts? > I assume that the student use this machine exclusively when he starts here. > > - kaz > > On 03/16/2015 01:25 PM, Allen, Benjamin S. wrote: >>> On Mar 16, 2015, at 1:15 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >>> >>> Hi Ben, >>> >>> Can we have an access to one of the teradimm block devices? >>> or can we move some of exFlash DIMMs to other system? >> We only have one system. We have a summer student this year under Kevin Harms thats slated todo research on the box, so it can be used till then. I'll need to be able to put back to normal before the student starts. >> >> Hostname is "wally.ftm.alcf.anl.gov". Let me know if you need sudo access. >> >>> John is interested in trying out teradimm. >>> I'm also interested in hacking the kernel module. >> Feel free to mess with the kernel module. No editing the BIOS settings on the box however. The BIOS appears to be vary fragile. It's very easy to get the machine into a state that doesn't boot. >> >>> BTW, do you know the cost of each DIMM? >>> do we have any other NVRAM devices in JLSE? >> We ordered the box last April. As a whole the server costs $29,611.00 with 8 exFlash DIMMs installed. The quote shows the list price as $4,949.00 for the 400GB exFlash DIMM. Our overall discounted price for the server was 44% off list price. >> >> No we don't have any other. I don't know of any other nvram product actually shipping yet. >> >> Ben >

1 0

Re: [JLSE-Admins] [JLSE-Discuss] IBM x3650 M4 with exFlash SSD DIMMs (Diablo Technologies Teradimm) - wally.ftm.alcf.anl.gov
by Kazutomo Yoshii 16 Mar '15

16 Mar '15

Ben, Thanks! John and I would like to have a sudo access, so that we can try something before the student starts working here. Do you know when the student starts? I assume that the student use this machine exclusively when he starts here. - kaz On 03/16/2015 01:25 PM, Allen, Benjamin S. wrote: >> On Mar 16, 2015, at 1:15 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >> >> Hi Ben, >> >> Can we have an access to one of the teradimm block devices? >> or can we move some of exFlash DIMMs to other system? > We only have one system. We have a summer student this year under Kevin Harms thats slated todo research on the box, so it can be used till then. I'll need to be able to put back to normal before the student starts. > > Hostname is "wally.ftm.alcf.anl.gov". Let me know if you need sudo access. > >> John is interested in trying out teradimm. >> I'm also interested in hacking the kernel module. > Feel free to mess with the kernel module. No editing the BIOS settings on the box however. The BIOS appears to be vary fragile. It's very easy to get the machine into a state that doesn't boot. > >> BTW, do you know the cost of each DIMM? >> do we have any other NVRAM devices in JLSE? > We ordered the box last April. As a whole the server costs $29,611.00 with 8 exFlash DIMMs installed. The quote shows the list price as $4,949.00 for the 400GB exFlash DIMM. Our overall discounted price for the server was 44% off list price. > > No we don't have any other. I don't know of any other nvram product actually shipping yet. > > Ben

1 0

Re: [JLSE-Admins] [JLSE-Discuss] IBM x3650 M4 with exFlash SSD DIMMs (Diablo Technologies Teradimm) - wally.ftm.alcf.anl.gov
by Allen, Benjamin S. 16 Mar '15

16 Mar '15

> On Mar 16, 2015, at 1:15 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: > > Hi Ben, > > Can we have an access to one of the teradimm block devices? > or can we move some of exFlash DIMMs to other system? We only have one system. We have a summer student this year under Kevin Harms thats slated todo research on the box, so it can be used till then. I'll need to be able to put back to normal before the student starts. Hostname is "wally.ftm.alcf.anl.gov". Let me know if you need sudo access. > > John is interested in trying out teradimm. > I'm also interested in hacking the kernel module. Feel free to mess with the kernel module. No editing the BIOS settings on the box however. The BIOS appears to be vary fragile. It's very easy to get the machine into a state that doesn't boot. > > BTW, do you know the cost of each DIMM? > do we have any other NVRAM devices in JLSE? We ordered the box last April. As a whole the server costs $29,611.00 with 8 exFlash DIMMs installed. The quote shows the list price as $4,949.00 for the 400GB exFlash DIMM. Our overall discounted price for the server was 44% off list price. No we don't have any other. I don't know of any other nvram product actually shipping yet. Ben

1 0

[JLSE-Admins] Fwd: [JLSE-Discuss] IBM x3650 M4 with exFlash SSD DIMMs (Diablo Technologies Teradimm) - wally.ftm.alcf.anl.gov
by Kazutomo Yoshii 16 Mar '15

16 Mar '15

Hi Ben, Can we have an access to one of the teradimm block devices? or can we move some of exFlash DIMMs to other system? John is interested in trying out teradimm. I'm also interested in hacking the kernel module. BTW, do you know the cost of each DIMM? do we have any other NVRAM devices in JLSE? - kaz -------- Forwarded Message -------- Subject: [JLSE-Discuss] IBM x3650 M4 with exFlash SSD DIMMs (Diablo Technologies Teradimm) - wally.ftm.alcf.anl.gov Date: Fri, 8 Aug 2014 16:52:48 +0000 From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> To: discuss(a)lists.jlse.anl.gov <discuss(a)lists.jlse.anl.gov> JLSE has installed a IBM x3650 with 8 400GB exFlash modules in JLSE as "wally". It only has a 1GbE connection currently. Initial benchmarks of the DIMMs are pretty good. Currently Kevin Harms is working on the machine. The devices are presented as block devices /dev/td{a..h}. The cli tool is "td-cli", and the kernel module is "teradimm". teradimm_ipmi_proxy communicates the status of the SSD DIMMs to the IMM/BMC. The DRAM and SSD DIMMs are all running at 1333Mhz for some reason even though all the modules installed are capable of 1600Mhz (1866Mhz for the DRAM). By IBM's published stats, this lowers throughput but actually increases IOPs performance a bit. I've found no way to force them into 1600Mhz in the BIOS. The SSD DIMMs have been updated with the latest firmware from IBM. I've also made sure the uEFI, IMM, and pDSA firmware versions are at least at the minimum levels to support exFlash DIMMs. I've also ensured the BIOS settings mentioned on http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=SERV-FLASHDM are all set correctly. It appears the devices are optimized for 4KB IO, throughput is even good at 4K, albeit a bit better at 64KB. It also appears they do better with only a single thread per DIMM. However this is based on about 30 minutes of benchmark with btest at this point. With a single thread at 4K latency is measuring at 8 usec, which is pretty insane for SSD. [bsallen@wally IBM]$ sudo /tmp/btest -b 4096 -T 2 S R /dev/tda /dev/tdb /dev/tdc /dev/tdd /dev/tde /dev/tdf /dev/tdg /dev/tdh IO Block size is 4096 Use sync IO mode Using the default 60 seconds time limit Use pure sequential IO Generate only reads 16 threads are ready, starting test at Thu Jul 31 22:19:58 2014 Subtotal (diff): 16 workers, 9.832 seconds (9.818), 741134.500 iops, avg latency 21 usec, bandwidth 2964528 KB/s, errors 00, ops 7287111 Subtotal (diff): 16 workers, 9.832 seconds (9.818), 1171 max_latency, hiccups levels: <1ms: 7287103 1ms: 8 2-10ms: 0 11-50ms: 0 51-100ms: 0 >100ms: 0 Subtotal (diff): 16 workers, 9.737 seconds (19.547), 734156.938 iops, avg latency 21 usec, bandwidth 2936624 KB/s, errors 00, ops 7148447 Subtotal (diff): 16 workers, 9.737 seconds (19.547), 1093 max_latency, hiccups levels: <1ms: 7148441 1ms: 6 2-10ms: 0 11-50ms: 0 51-100ms: 0 >100ms: 0 Subtotal (diff): 16 workers, 9.735 seconds (29.274), 737060.625 iops, avg latency 21 usec, bandwidth 2948240 KB/s, errors 00, ops 7175456 Subtotal (diff): 16 workers, 9.735 seconds (29.274), 1021 max_latency, hiccups levels: <1ms: 7175453 1ms: 3 2-10ms: 0 11-50ms: 0 51-100ms: 0 >100ms: 0 Subtotal (diff): 16 workers, 9.735 seconds (39.001), 736089.625 iops, avg latency 21 usec, bandwidth 2944352 KB/s, errors 00, ops 7166093 Subtotal (diff): 16 workers, 9.735 seconds (39.001), 7815 max_latency, hiccups levels: <1ms: 7166084 1ms: 5 2-10ms: 4 11-50ms: 0 51-100ms: 0 >100ms: 0 Subtotal (diff): 16 workers, 9.756 seconds (48.740), 693506.188 iops, avg latency 23 usec, bandwidth 2774016 KB/s, errors 00, ops 6765596 Subtotal (diff): 16 workers, 9.756 seconds (48.740), 8052 max_latency, hiccups levels: <1ms: 6765589 1ms: 3 2-10ms: 4 11-50ms: 0 51-100ms: 0 >100ms: 0 Total: 58.554 seconds, 722776.438 iops, avg latency 22 usec, bandwidth 2891104 KB/s, errors 0, total ops 42321669 Total: 58.554 seconds, 8052 max_latency, hiccups levels: <1ms: 42321608 1ms: 43 2-10ms: 18 11-50ms: 0 51-100ms: 0 >100ms: 0 Test is done at Thu Jul 31 22:20:59 2014 [bsallen@wally IBM]$ sudo /tmp/btest -b 4096 -T 2 S W /dev/tda /dev/tdb /dev/tdc /dev/tdd /dev/tde /dev/tdf /dev/tdg /dev/tdh IO Block size is 4096 Use sync IO mode Using the default 60 seconds time limit Use pure sequential IO Generate only writes 16 threads are ready, starting test at Thu Jul 31 22:22:03 2014 Subtotal (diff): 16 workers, 9.906 seconds (9.901), 554011.438 iops, avg latency 28 usec, bandwidth 2216032 KB/s, errors 00, ops 5488264 Subtotal (diff): 16 workers, 9.906 seconds (9.901), 482 max_latency, hiccups levels: <1ms: 5488264 1ms: 0 2-10ms: 0 11-50ms: 0 51-100ms: 0 >100ms: 0 Subtotal (diff): 16 workers, 9.814 seconds (19.711), 546904.312 iops, avg latency 29 usec, bandwidth 2187616 KB/s, errors 00, ops 5367422 Subtotal (diff): 16 workers, 9.814 seconds (19.711), 1772 max_latency, hiccups levels: <1ms: 5367420 1ms: 2 2-10ms: 0 11-50ms: 0 51-100ms: 0 >100ms: 0 Subtotal (diff): 16 workers, 9.812 seconds (29.525), 553523.938 iops, avg latency 28 usec, bandwidth 2214080 KB/s, errors 00, ops 5431129 Subtotal (diff): 16 workers, 9.812 seconds (29.525), 1032 max_latency, hiccups levels: <1ms: 5431127 1ms: 2 2-10ms: 0 11-50ms: 0 51-100ms: 0 >100ms: 0 Subtotal (diff): 16 workers, 9.811 seconds (39.331), 553460.688 iops, avg latency 28 usec, bandwidth 2213840 KB/s, errors 00, ops 5430267 Subtotal (diff): 16 workers, 9.811 seconds (39.331), 549 max_latency, hiccups levels: <1ms: 5430267 1ms: 0 2-10ms: 0 11-50ms: 0 51-100ms: 0 >100ms: 0 Subtotal (diff): 16 workers, 9.812 seconds (49.139), 553227.312 iops, avg latency 28 usec, bandwidth 2212896 KB/s, errors 00, ops 5428196 Subtotal (diff): 16 workers, 9.812 seconds (49.139), 544 max_latency, hiccups levels: <1ms: 5428196 1ms: 0 2-10ms: 0 11-50ms: 0 51-100ms: 0 >100ms: 0 Total: 58.970 seconds, 551339.438 iops, avg latency 29 usec, bandwidth 2205344 KB/s, errors 0, total ops 32512339 Total: 58.970 seconds, 1804 max_latency, hiccups levels: <1ms: 32512333 1ms: 6 2-10ms: 0 11-50ms: 0 51-100ms: 0 >100ms: 0 Test is done at Thu Jul 31 22:23:04 2014 References: http://www.diablo-technologies.com/ http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=SERV-FLASHDM exFlash DIMM v1.4.7 software including kernel module source: http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5095918 _______________________________________________ Discuss mailing list Discuss(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/discuss

1 0

[JLSE-Admins] [Fail2Ban] SSH: banned 213.182.43.222 on
by Fail2Ban 16 Mar '15

16 Mar '15

The IP 213.182.43.222 has just been banned by Fail2Ban after 18 attempts against SSH. Latest lines containing IP:213.182.43.222 in /var/log/syslog/sshd.log Mar 16 07:23:52 ftlogin1 sshd[9655]: Received disconnect from 213.182.43.222: 3: com.jcraft.jsch.JSchException: Auth cancel Mar 16 07:23:54 ftlogin1 sshd[9661]: Postponed keyboard-interactive for root from 213.182.43.222 port 28583 ssh2 Mar 16 07:23:56 ftlogin1 sshd[9658]: Failed keyboard-interactive/pam for root from 213.182.43.222 port 28583 ssh2 Mar 16 07:23:56 ftlogin1 sshd[9661]: Received disconnect from 213.182.43.222: 3: com.jcraft.jsch.JSchException: Auth cancel Mar 16 07:23:56 ftlogin1 sshd[9664]: Invalid user guest from 213.182.43.222 Mar 16 07:23:56 ftlogin1 sshd[9667]: Postponed keyboard-interactive for invalid user guest from 213.182.43.222 port 29456 ssh2 Mar 16 07:23:58 ftlogin1 sshd[9664]: Failed keyboard-interactive/pam for invalid user guest from 213.182.43.222 port 29456 ssh2 Mar 16 07:23:58 ftlogin1 sshd[9667]: Postponed keyboard-interactive for invalid user guest from 213.182.43.222 port 29456 ssh2 Mar 16 07:23:59 ftlogin1 sshd[9664]: Failed keyboard-interactive/pam for invalid user guest from 213.182.43.222 port 29456 ssh2 Mar 16 07:23:59 ftlogin1 sshd[9667]: Received disconnect from 213.182.43.222: 3: com.jcraft.jsch.JSchException: Auth cancel - Fail2ban on

1 0

[JLSE-Admins] [Fail2Ban] SSH: banned 213.182.43.222 on
by Fail2Ban 16 Mar '15

16 Mar '15

The IP 213.182.43.222 has just been banned by Fail2Ban after 18 attempts against SSH. Latest lines containing IP:213.182.43.222 in /var/log/syslog/sshd.log Mar 16 07:23:53 ftlogin2 sshd[4695]: Received disconnect from 213.182.43.222: 3: com.jcraft.jsch.JSchException: Auth cancel Mar 16 07:23:55 ftlogin2 sshd[4701]: Postponed keyboard-interactive for root from 213.182.43.222 port 28606 ssh2 Mar 16 07:23:56 ftlogin2 sshd[4698]: Failed keyboard-interactive/pam for root from 213.182.43.222 port 28606 ssh2 Mar 16 07:23:56 ftlogin2 sshd[4701]: Received disconnect from 213.182.43.222: 3: com.jcraft.jsch.JSchException: Auth cancel Mar 16 07:23:57 ftlogin2 sshd[4704]: Invalid user guest from 213.182.43.222 Mar 16 07:23:57 ftlogin2 sshd[4707]: Postponed keyboard-interactive for invalid user guest from 213.182.43.222 port 29514 ssh2 Mar 16 07:23:58 ftlogin2 sshd[4704]: Failed keyboard-interactive/pam for invalid user guest from 213.182.43.222 port 29514 ssh2 Mar 16 07:23:58 ftlogin2 sshd[4707]: Postponed keyboard-interactive for invalid user guest from 213.182.43.222 port 29514 ssh2 Mar 16 07:23:59 ftlogin2 sshd[4704]: Failed keyboard-interactive/pam for invalid user guest from 213.182.43.222 port 29514 ssh2 Mar 16 07:23:59 ftlogin2 sshd[4707]: Received disconnect from 213.182.43.222: 3: com.jcraft.jsch.JSchException: Auth cancel - Fail2ban on

1 0

Re: [JLSE-Admins] Meeting this morning
by Rajeev Thakur 13 Mar '15

13 Mar '15

Nothing for me. I am in a meeting in another building in any case. We can switch to once a month and can meet earlier if a need arises. Rajeev On Mar 13, 2015, at 9:19 AM, Kalyan Kumaran <kumaran(a)anl.gov> wrote: > Any agenda topics that requires a F2F meeting this morning? > > Should we switch to meeting once a month? Maybe last Friday of the month? > > Thanks, > Kumar > _______________________________________________ > Admins mailing list > Admins(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/admins

1 0

Re: [JLSE-Admins] Meeting this morning
by Kazutomo Yoshii 13 Mar '15

13 Mar '15

I have no big agenda for all hand face-to-face meeting, while I have a few specific agenda. - narrowing down a spec of a NUMA machine. Ben is helping. - Is anyone interested in a steal mode company DSSD (acquired by EMC recently)? http://www.emc.com/about/news/press/2014/20140505-06.htm NDA is required to know details as usual. - kaz On 03/13/2015 09:19 AM, Kalyan Kumaran wrote: > Any agenda topics that requires a F2F meeting this morning? > > Should we switch to meeting once a month? Maybe last Friday of the month? > > Thanks, > Kumar > _______________________________________________ > Admins mailing list > Admins(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/admins

1 0

[JLSE-Admins] Meeting this morning
by Kalyan Kumaran 13 Mar '15

13 Mar '15

Any agenda topics that requires a F2F meeting this morning? Should we switch to meeting once a month? Maybe last Friday of the month? Thanks, Kumar

1 0