rawhide_k/blog.neet.works
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
blog.neet.works/public/posts/watercooling-homelab/index.html
rawhide kobayashi f0665cec13
All checks were successful
blog.neet.works deployment / deploy_mamizou (push) Successful in 5s
Details
maxcloudon pcie riser post + keywords for watercooling post
2025-01-19 22:37:57 -06:00

180 lines
100 KiB
HTML
Raw Blame History

<!doctype html><html lang=en dir=ltr class=scroll-smooth data-default-appearance=dark data-auto-appearance=false><head><meta charset=utf-8><meta http-equiv=content-language content="en"><meta name=viewport content="width=device-width,initial-scale=1"><meta http-equiv=X-UA-Compatible content="ie=edge"><title>Watercooling My Homelab &#183; N.E.E.T. Works</title>
<meta name=title content="Watercooling My Homelab &#183; N.E.E.T. Works"><meta name=description content="Watercooling for my homelab with a custom, leak-resistant controller and monitoring!"><meta name=keywords content="alphacool,arduino,intel,nvidia,supermicro,aqua computer,"><link rel=canonical href=https://blog.neet.works/posts/watercooling-homelab/><link type=text/css rel=stylesheet href=/css/main.bundle.min.40e00100ae5ec4637cecd46ba4ae5f8db39bb83da61b5efe4900676fc0f4c10ac17e7da11e467fb8f1f39dfa5a5070c05bba06b4c05c629c2d389f4263d4c633.css integrity="sha512-QOABAK5exGN87NRrpK5fjbObuD2mG17+SQBnb8D0wQrBfn2hHkZ/uPHznfpaUHDAW7oGtMBcYpwtOJ9CY9TGMw=="><script type=text/javascript src=/js/appearance.min.516a16745bea5a9bd011138d254cc0fd3973cd55ce6e15f3dec763e7c7c2c7448f8fe7b54cca811cb821b0c7e12cd161caace1dd794ac3d34d40937cbcc9ee12.js integrity="sha512-UWoWdFvqWpvQERONJUzA/TlzzVXObhXz3sdj58fCx0SPj+e1TMqBHLghsMfhLNFhyqzh3XlKw9NNQJN8vMnuEg=="></script><script defer type=text/javascript id=script-bundle src=/js/main.bundle.min.a2d78d78672e549fbfc972ece871725b5478ba0b65708dda20cb97ab80a865eae6d247e1b05a4aec6ebbf78647ec3233bad8b2609ed98eee53cd58aa17128bc7.js integrity="sha512-oteNeGcuVJ+/yXLs6HFyW1R4ugtlcI3aIMuXq4CoZerm0kfhsFpK7G6794ZH7DIzutiyYJ7Zju5TzViqFxKLxw==" data-copy data-copied></script><link rel=apple-touch-icon sizes=180x180 href=/apple-touch-icon.png><link rel=icon type=image/png sizes=32x32 href=/favicon-32x32.png><link rel=icon type=image/png sizes=16x16 href=/favicon-16x16.png><link rel=manifest href=/site.webmanifest><meta property="og:url" content="https://blog.neet.works/posts/watercooling-homelab/"><meta property="og:site_name" content="N.E.E.T. Works"><meta property="og:title" content="Watercooling My Homelab"><meta property="og:description" content="Watercooling for my homelab with a custom, leak-resistant controller and monitoring!"><meta property="og:locale" content="en"><meta property="og:type" content="article"><meta property="article:section" content="posts"><meta property="article:published_time" content="2025-01-16T05:56:41+00:00"><meta property="article:modified_time" content="2025-01-16T05:56:41+00:00"><meta property="article:tag" content="Alphacool"><meta property="article:tag" content="Arduino"><meta property="article:tag" content="Intel"><meta property="article:tag" content="Nvidia"><meta property="article:tag" content="Supermicro"><meta property="article:tag" content="Aqua Computer"><meta property="og:image" content="https://blog.neet.works/posts/watercooling-homelab/featured.webp"><meta name=twitter:card content="summary_large_image"><meta name=twitter:image content="https://blog.neet.works/posts/watercooling-homelab/featured.webp"><meta name=twitter:title content="Watercooling My Homelab"><meta name=twitter:description content="Watercooling for my homelab with a custom, leak-resistant controller and monitoring!"><script type=application/ld+json>[{"@context":"https://schema.org","@type":"Article","articleSection":"Posts","name":"Watercooling My Homelab","headline":"Watercooling My Homelab","description":"Watercooling for my homelab with a custom, leak-resistant controller and monitoring!","inLanguage":"en","url":"https:\/\/blog.neet.works\/posts\/watercooling-homelab\/","author":{"@type":"Person","name":"rawhide kobayashi"},"copyrightYear":"2025","dateCreated":"2025-01-16T05:56:41\u002b00:00","datePublished":"2025-01-16T05:56:41\u002b00:00","dateModified":"2025-01-16T05:56:41\u002b00:00","keywords":["Aqua Computer LEAKSHIELD alternative","Custom homelab server cooling","Custom server rack watercooling system","DIY Linux-compatible water cooling","DIY server cooling","GPU watercooling for ML tasks","Homelab rack-mounted watercooling","homelab water cooling","Leak detection for custom water cooling loops","Leak-resistant water cooling systems","leak-resistant watercooling","server water cooling","server water cooling system","Using Arduino for custom watercooling controllers","Watercooling a server rack","Watercooling with vacuum leak detection","Leak prevention in water cooling systems"],"mainEntityOfPage":"true","wordCount":"5240"}]</script><meta name=author content="rawhide kobayashi"><link href=mailto:rawhide@neet.works rel=me><link href=https://git.neet.works/rawhide_k rel=me><link href=https://github.com/rawhide-kobayashi rel=me><script src=/lib/jquery/jquery.slim.min.b0dca576e87d7eaa5850ae4e61759c065786cdb6489d68fcc82240539eebd5da522bdb4fda085ffd245808c8fe2acb2516408eb774ef26b5f6015fc6737c0ea8.js integrity="sha512-sNylduh9fqpYUK5OYXWcBleGzbZInWj8yCJAU57r1dpSK9tP2ghf/SRYCMj+KsslFkCOt3TvJrX2AV/Gc3wOqA=="></script><script defer src=/lib/packery/packery.pkgd.min.js integrity></script><script type=text/javascript src=/js/shortcodes/gallery.min.5fddb0ac85fed097f7b3630b6616e67caf1eea7ca23a42a6f7fc645f561306e7dabaef0a4eda37c74fb627ae3c97e11cb39b8895a4c839816632940e4a482508.js integrity="sha512-X92wrIX+0Jf3s2MLZhbmfK8e6nyiOkKm9/xkX1YTBufauu8KTto3x0+2J648l+Ecs5uIlaTIOYFmMpQOSkglCA=="></script><link rel=stylesheet href=/lib/lite-youtube-embed/lite-yt-embed.css integrity><script src=/lib/lite-youtube-embed/lite-yt-embed.58a8a22aed9d1cd05780869df4b0d87972fcd144a9eaff8f9a1453e58e6521c5a89ccd80bdb9f48e25ae5ee87b98bd9de51b14dd9988c60e4c89630433292084.js integrity="sha512-WKiiKu2dHNBXgIad9LDYeXL80USp6v+PmhRT5Y5lIcWonM2Avbn0jiWuXuh7mL2d5RsU3ZmIxg5MiWMEMykghA=="></script><meta name=theme-color><script type=text/javascript src=/js/panzoom.min.b48202c3408ba8ef63ef9a6cc583e9d7ddb6574828cf79a4aaefe3c3964d0098ad77da46fdd5dddb66314ca975f6b9562093fa227ef51db7eba6d640facf448c.js integrity="sha512-tIICw0CLqO9j75psxYPp1922V0goz3mkqu/jw5ZNAJitd9pG/dXd22YxTKl19rlWIJP6In71HbfrptZA+s9EjA=="></script><script type=text/javascript src=/js/panzoom-util.687cf1af7ca1d075a5ce49ea7cef103a94c400423cc5b30927bf4db06c6a8ae0b8ee6f2b2bf501be099039b2f4ac5f4b348ed4af0429559ebb0743c54f051d5a.js integrity="sha512-aHzxr3yh0HWlzknqfO8QOpTEAEI8xbMJJ79NsGxqiuC47m8rK/UBvgmQObL0rF9LNI7UrwQpVZ67B0PFTwUdWg=="></script></head><body class="flex flex-col h-screen px-6 m-auto text-lg leading-7 max-w-7xl bg-neutral text-neutral-900 dark:bg-neutral-800 dark:text-neutral sm:px-14 md:px-24 lg:px-32 scrollbar-thin scrollbar-track-neutral-200 scrollbar-thumb-neutral-400 dark:scrollbar-track-neutral-800 dark:scrollbar-thumb-neutral-600"><div id=the-top class="absolute flex self-center"><a class="px-3 py-1 text-sm -translate-y-8 rounded-b-lg bg-primary-200 focus:translate-y-0 dark:bg-neutral-600" href=#main-content><span class="font-bold text-primary-600 ltr:pr-2 rtl:pl-2 dark:text-primary-400">&darr;</span>Skip to main content</a></div><div style=padding-left:0;padding-right:0;padding-top:2px;padding-bottom:3px class="main-menu flex items-center justify-between px-4 py-6 sm:px-6 md:justify-start space-x-3"><div class="flex flex-1 items-center justify-between"><nav class="flex space-x-3"><a href=/ class="text-base font-medium text-gray-500 hover:text-gray-900">N.E.E.T. Works</a></nav><nav class="hidden md:flex items-center space-x-5 md:ml-12 h-12"><a href=/posts/ class="flex items-center text-gray-500 hover:text-primary-600 dark:hover:text-primary-400"><p class="text-base font-medium" title>Blog</p></a><a href=/categories/ class="flex items-center text-gray-500 hover:text-primary-600 dark:hover:text-primary-400"><p class="text-base font-medium" title>Categories</p></a><a href=/tags/ class="flex items-center text-gray-500 hover:text-primary-600 dark:hover:text-primary-400"><p class="text-base font-medium" title>Tags</p></a><button id=search-button aria-label=Search class="text-base hover:text-primary-600 dark:hover:text-primary-400" title>
<span class="relative block icon"><svg aria-hidden="true" focusable="false" data-prefix="fas" data-icon="search" class="svg-inline--fa fa-search fa-w-16" role="img" viewBox="0 0 512 512"><path fill="currentcolor" d="M505 442.7 405.3 343c-4.5-4.5-10.6-7-17-7H372c27.6-35.3 44-79.7 44-128C416 93.1 322.9.0 208 0S0 93.1.0 208s93.1 208 208 208c48.3.0 92.7-16.4 128-44v16.3c0 6.4 2.5 12.5 7 17l99.7 99.7c9.4 9.4 24.6 9.4 33.9.0l28.3-28.3c9.4-9.4 9.4-24.6.1-34zM208 336c-70.7.0-128-57.2-128-128 0-70.7 57.2-128 128-128 70.7.0 128 57.2 128 128 0 70.7-57.2 128-128 128z"/></svg></span></button></nav><div class="flex md:hidden items-center space-x-5 md:ml-12 h-12"><span></span>
<button id=search-button-mobile aria-label=Search class="text-base hover:text-primary-600 dark:hover:text-primary-400" title>
<span class="relative block icon"><svg aria-hidden="true" focusable="false" data-prefix="fas" data-icon="search" class="svg-inline--fa fa-search fa-w-16" role="img" viewBox="0 0 512 512"><path fill="currentcolor" d="M505 442.7 405.3 343c-4.5-4.5-10.6-7-17-7H372c27.6-35.3 44-79.7 44-128C416 93.1 322.9.0 208 0S0 93.1.0 208s93.1 208 208 208c48.3.0 92.7-16.4 128-44v16.3c0 6.4 2.5 12.5 7 17l99.7 99.7c9.4 9.4 24.6 9.4 33.9.0l28.3-28.3c9.4-9.4 9.4-24.6.1-34zM208 336c-70.7.0-128-57.2-128-128 0-70.7 57.2-128 128-128 70.7.0 128 57.2 128 128 0 70.7-57.2 128-128 128z"/></svg></span></button></div></div><div class="-my-2 -mr-2 md:hidden"><label id=menu-button class=block><div class="cursor-pointer hover:text-primary-600 dark:hover:text-primary-400"><span class="relative block icon"><svg viewBox="0 0 448 512"><path fill="currentcolor" d="M0 96C0 78.33 14.33 64 32 64H416c17.7.0 32 14.33 32 32 0 17.7-14.3 32-32 32H32C14.33 128 0 113.7.0 96zM0 256c0-17.7 14.33-32 32-32H416c17.7.0 32 14.3 32 32s-14.3 32-32 32H32c-17.67.0-32-14.3-32-32zM416 448H32c-17.67.0-32-14.3-32-32s14.33-32 32-32H416c17.7.0 32 14.3 32 32s-14.3 32-32 32z"/></svg></span></div><div id=menu-wrapper style=padding-top:5px class="fixed inset-0 z-30 invisible w-screen h-screen m-0 overflow-auto transition-opacity opacity-0 cursor-default bg-neutral-100/50 backdrop-blur-sm dark:bg-neutral-900/50"><ul class="flex space-y-2 mt-3 flex-col items-end w-full px-6 py-6 mx-auto overflow-visible list-none ltr:text-right rtl:text-left max-w-7xl"><li id=menu-close-button><span class="cursor-pointer inline-block align-text-bottom hover:text-primary-600 dark:hover:text-primary-400"><span class="relative block icon"><svg viewBox="0 0 320 512"><path fill="currentcolor" d="M310.6 361.4c12.5 12.5 12.5 32.75.0 45.25C304.4 412.9 296.2 416 288 416s-16.38-3.125-22.62-9.375L160 301.3 54.63 406.6C48.38 412.9 40.19 416 32 416S15.63 412.9 9.375 406.6c-12.5-12.5-12.5-32.75.0-45.25l105.4-105.4L9.375 150.6c-12.5-12.5-12.5-32.75.0-45.25s32.75-12.5 45.25.0L160 210.8l105.4-105.4c12.5-12.5 32.75-12.5 45.25.0s12.5 32.75.0 45.25l-105.4 105.4L310.6 361.4z"/></svg></span></span></li><li class=mt-1><a href=/posts/ class="flex items-center text-gray-500 hover:text-primary-600 dark:hover:text-primary-400"><p class="text-bg font-bg" title>Blog</p></a></li><li class=mt-1><a href=/categories/ class="flex items-center text-gray-500 hover:text-primary-600 dark:hover:text-primary-400"><p class="text-bg font-bg" title>Categories</p></a></li><li class=mt-1><a href=/tags/ class="flex items-center text-gray-500 hover:text-primary-600 dark:hover:text-primary-400"><p class="text-bg font-bg" title>Tags</p></a></li></ul></div></label></div></div><div class="relative flex flex-col grow"><main id=main-content class=grow><article><div class="fixed inset-x-0 top-0 h-[800px] single_hero_background nozoom" style=background-image:url(/posts/watercooling-homelab/featured_hu11504899310106128621.webp)><div class="absolute inset-0 bg-gradient-to-t from-neutral dark:from-neutral-800 to-transparent mix-blend-normal"></div><div class="absolute inset-0 opacity-60 bg-gradient-to-t from-neutral dark:from-neutral-800 to-neutral-100 dark:to-neutral-800 mix-blend-normal"></div></div><section class="flex flex-col max-w-full mt-0 prose dark:prose-invert lg:flex-row"><div class="order-first lg:ml-auto px-0 lg:order-last ltr:lg:pl-8 rtl:lg:pr-8"><div class="toc ltr:pl-5 rtl:pr-5 print:hidden lg:sticky lg:top-10"><details open id=TOCView class="toc-right mt-0 overflow-y-scroll overscroll-contain scrollbar-thin scrollbar-track-neutral-200 scrollbar-thumb-neutral-400 dark:scrollbar-track-neutral-800 dark:scrollbar-thumb-neutral-600 rounded-lg ltr:-ml-5 ltr:pl-5 rtl:-mr-5 rtl:pr-5 hidden lg:block"><summary class="block py-1 text-lg font-semibold cursor-pointer bg-neutral-100 text-neutral-800 ltr:-ml-5 ltr:pl-5 rtl:-mr-5 rtl:pr-5 dark:bg-neutral-700 dark:text-neutral-100 lg:hidden">Table of Contents</summary><div class="min-w-[220px] py-2 border-dotted ltr:-ml-5 ltr:border-l ltr:pl-5 rtl:-mr-5 rtl:border-r rtl:pr-5 dark:border-neutral-600"><nav id=TableOfContents><ul><li><a href=#overview>Overview</a></li><li><a href=#the-hardware>The Hardware</a><ul><li><a href=#off-the-shelf>Off-the-shelf</a><ul><li><a href=#general-details>General Details</a></li><li><a href=#cpus>CPUs</a></li><li><a href=#gpus>GPUs</a></li></ul></li><li><a href=#diy-time>DIY Time</a><ul><li><a href=#putting-new-life-into-an-imac-psu>Putting New Life into an iMac PSU</a></li><li><a href=#measuring-vacuum>Measuring vacuum</a></li><li><a href=#other-stuff>Other stuff</a></li></ul></li></ul></li><li><a href=#the-software>The Software</a></li><li><a href=#other-thoughts>Other Thoughts?</a></li></ul></nav></div></details><details class="toc-inside mt-0 overflow-hidden rounded-lg ltr:-ml-5 ltr:pl-5 rtl:-mr-5 rtl:pr-5 lg:hidden"><summary class="py-1 text-lg font-semibold cursor-pointer bg-neutral-100 text-neutral-800 ltr:-ml-5 ltr:pl-5 rtl:-mr-5 rtl:pr-5 dark:bg-neutral-700 dark:text-neutral-100 lg:hidden">Table of Contents</summary><div class="py-2 border-dotted border-neutral-300 ltr:-ml-5 ltr:border-l ltr:pl-5 rtl:-mr-5 rtl:border-r rtl:pr-5 dark:border-neutral-600"><nav id=TableOfContents><ul><li><a href=#overview>Overview</a></li><li><a href=#the-hardware>The Hardware</a><ul><li><a href=#off-the-shelf>Off-the-shelf</a><ul><li><a href=#general-details>General Details</a></li><li><a href=#cpus>CPUs</a></li><li><a href=#gpus>GPUs</a></li></ul></li><li><a href=#diy-time>DIY Time</a><ul><li><a href=#putting-new-life-into-an-imac-psu>Putting New Life into an iMac PSU</a></li><li><a href=#measuring-vacuum>Measuring vacuum</a></li><li><a href=#other-stuff>Other stuff</a></li></ul></li></ul></li><li><a href=#the-software>The Software</a></li><li><a href=#other-thoughts>Other Thoughts?</a></li></ul></nav></div></details><script>var margin=200,marginError=50;(function(){var t=$(window),e=$("#TOCView"),s=e.height();function n(){var n=t.height()-margin;s>=n?(e.css("overflow-y","scroll"),e.css("max-height",n+marginError+"px")):(e.css("overflow-y","hidden"),e.css("max-height","9999999px"))}t.on("resize",n),$(document).ready(n)})()</script></div></div><div class="min-w-0 min-h-0 max-w-fit"><header id=single_header class="mt-5 max-w-prose"><h1 class="mt-0 text-4xl font-extrabold cool-title">Watercooling My Homelab</h1><div class=vari-flex><img class="!mt-0 !mb-0 h-24 w-24 rounded-full ltr:mr-4 rtl:ml-4" width=96 height=96 alt="rawhide kobayashi" src=/img/blowfish_logo_hu536110971430689468.webp><div class=place-self-center><div class="text-[0.6rem] uppercase leading-3 text-neutral-500 dark:text-neutral-400">Author</div><div class="font-semibold leading-6 text-neutral-800 dark:text-neutral-300">rawhide kobayashi</div><div class="text-sm text-neutral-700 dark:text-neutral-400">ᗜ⩊ᗜ</div><div class="text-2xl sm:text-lg"><div class="flex flex-wrap text-neutral-400 dark:text-neutral-500"><a class="px-1 hover:text-primary-700 dark:hover:text-primary-400" href=mailto:rawhide@neet.works target=_blank aria-label=Email rel="me noopener noreferrer"><span class="inline-block align-text-bottom"><span class="relative block icon"><svg viewBox="0 0 512 512"><path fill="currentcolor" d="M207.8 20.73c-93.45 18.32-168.7 93.66-187 187.1-27.64 140.9 68.65 266.2 199.1 285.1 19.01 2.888 36.17-12.26 36.17-31.49l1e-4-.6631c0-15.74-11.44-28.88-26.84-31.24-84.35-12.98-149.2-86.13-149.2-174.2.0-102.9 88.61-185.5 193.4-175.4 91.54 8.869 158.6 91.25 158.6 183.2v16.16c0 22.09-17.94 40.05-40 40.05s-40.01-17.96-40.01-40.05v-120.1c0-8.847-7.161-16.02-16.01-16.02l-31.98.0036c-7.299.0-13.2 4.992-15.12 11.68-24.85-12.15-54.24-16.38-86.06-5.106-38.75 13.73-68.12 48.91-73.72 89.64-9.483 69.01 43.81 128 110.9 128 26.44.0 50.43-9.544 69.59-24.88 24 31.3 65.23 48.69 109.4 37.49C465.2 369.3 496 324.1 495.1 277.2V256.3c0-149.2-133.9-265.632-287.3-235.57zM239.1 304.3c-26.47.0-48-21.56-48-48.05s21.53-48.05 48-48.05 48 21.56 48 48.05-20.6 48.05-48 48.05z"/></svg>
</span></span></a><a class="px-1 hover:text-primary-700 dark:hover:text-primary-400" href=https://git.neet.works/rawhide_k target=_blank aria-label=Gitea rel="me noopener noreferrer"><span class="inline-block align-text-bottom"><span class="relative block icon"><svg viewBox="5.67 143.05 628.65 387.55"><path fill="currentcolor" d="M115.912 143.075c-6.462.0-13.762.525-22.012 2.325-8.7 1.8-33.5 7.4-53.8 26.9C-4.9 212.4 6.6 276.2 8 285.8c1.7 11.7 6.9 44.2 31.7 72.5 45.8 56.1 144.4 54.8 144.4 54.8s12.1 28.9 30.6 55.5c25 33.1 50.7 58.9 75.7 62 63 0 188.9-.1 188.9-.1s12 .1 28.3-10.3c14-8.5 26.5-23.4 26.5-23.4S547 483 565 451.5c5.5-9.7 10.1-19.1 14.1-28 0 0 55.2-117.1 55.2-231.1-1.1-34.5-9.6-40.6-11.6-42.6-4.1-4.1-9.6-4-9.6-4s-117.2 6.6-177.9 8c-13.267.3-26.433.597-39.5.697l.1 117.002s57.4 24.202 83.1 40.102c3.7 2.3 10.2 6.798 12.9 14.398 2.1 6.1 2 13.101-1 19.301l-61 126.9c-6.2 12.7-21.4 18.1-33.9 12l-126.9-61c-12.5-6-17.9-21.2-11.8-33.8l61-126.9c6-12.5 21.2-17.9 33.8-11.8a5290.322 5290.322.0 0027 12.954c0-36.449-.1-109.053-.1-109.053-29 .4-89.2-2.201-89.2-2.201s-141.4-7.1-156.8-8.5c-4.9-.3-10.525-.825-16.988-.825zm12.188 48.026s7.1 59.399 15.7 94.199c7.2 29.2 24.8 77.7 24.8 77.7s-26.1-3.1-43-9.1c-25.9-8.5-36.9-18.7-36.9-18.7S69.6 321.8 60 295.4c-16.5-44.2-1.4-71.2-1.4-71.2s8.4-22.5 38.5-30c13.8-3.7 31-3.1 31-3.1z"/><path fill="currentcolor" d="M326.8 380.1c-8.2.1-15.4 5.8-17.3 13.8-1.9 8 2 16.3 9.1 20 7.7 4 17.5 1.8 22.7-5.4 5.1-7.1 4.3-16.9-1.8-23.1l24-49.1c1.5.1 3.7.2 6.2-.5 4.1-.9 7.1-3.6 7.1-3.6 4.2 1.8 8.6 3.8 13.2 6.1 4.8 2.4 9.3 4.9 13.4 7.3.9.5 1.8 1.1 2.8 1.9 1.6 1.3 3.4 3.1 4.7 5.5 1.9 5.5-1.9 14.9-1.9 14.9-2.3 7.6-18.4 40.6-18.4 40.6-8.1-.2-15.3 5-17.7 12.5-2.6 8.1 1.1 17.3 8.9 21.3 7.8 4 17.4 1.7 22.5-5.3 5-6.8 4.6-16.3-1.1-22.6 1.9-3.7 3.7-7.4 5.6-11.3 5-10.4 13.5-30.4 13.5-30.4.9-1.7 5.7-10.3 2.7-21.3-2.5-11.4-12.6-16.7-12.6-16.7-12.2-7.9-29.2-15.2-29.2-15.2s0-4.1-1.1-7.1c-1.1-3.1-2.8-5.1-3.9-6.3 4.7-9.7 9.4-19.3 14.1-29-4.1-2-8.1-4-12.2-6.1-4.8 9.8-9.7 19.7-14.5 29.5-6.7-.1-12.9 3.5-16.1 9.4-3.4 6.3-2.7 14.1 1.9 19.8l-24.6 50.4z"/></svg>
</span></span></a><a class="px-1 hover:text-primary-700 dark:hover:text-primary-400" href=https://github.com/rawhide-kobayashi target=_blank aria-label=Github rel="me noopener noreferrer"><span class="inline-block align-text-bottom"><span class="relative block icon"><svg viewBox="0 0 496 512"><path fill="currentcolor" d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6.0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6.0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3.0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1.0-6.2-.3-40.4-.3-61.4.0.0-70 15-84.7-29.8.0.0-11.4-29.1-27.8-36.6.0.0-22.9-15.7 1.6-15.4.0.0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5.0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9.0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4.0 33.7-.3 75.4-.3 83.6.0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6.0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9.0-6.2-1.4-2.3-4-3.3-5.6-2z"/></svg></span></span></a></div></div></div><div style=padding-left:8px class="mt-1 mb-6 text-base text-neutral-500 dark:text-neutral-400 print:hidden max-w-fit"><div class="flex flex-row flex-wrap items-center"><time datetime=2025-01-16T05:56:41+00:00>January 16 2025</time><span class="px-2 text-primary-500">&#183;</span><span>5240 words</span></div><hr><div class="flex flex-row flex-wrap items-center"><span style=margin-top:.5rem class=mr-2 onclick='return window.open("/categories/homelab/","_self"),!1'><span class=flex style=cursor:pointer><span class="rounded-md border border-primary-400 px-1 py-[1px] text-xs font-normal">Homelab
</span></span></span><span style=margin-top:.5rem class=mr-2 onclick='return window.open("/categories/servers/","_self"),!1'><span class=flex style=cursor:pointer><span class="rounded-md border border-primary-400 px-1 py-[1px] text-xs font-normal">Servers
</span></span></span><span style=margin-top:.5rem class=mr-2 onclick='return window.open("/categories/watercooling/","_self"),!1'><span class=flex style=cursor:pointer><span class="rounded-md border border-primary-400 px-1 py-[1px] text-xs font-normal">Watercooling
</span></span></span><span style=margin-top:.5rem class=mr-2 onclick='return window.open("/categories/diy/","_self"),!1'><span class=flex style=cursor:pointer><span class="rounded-md border border-primary-400 px-1 py-[1px] text-xs font-normal">Diy
</span></span></span><span style=margin-top:.5rem class=mr-2 onclick='return window.open("/tags/alphacool/","_self"),!1'><span class=flex style=cursor:pointer><span class="rounded-md border border-primary-400 px-1 py-[1px] text-xs font-normal">Alphacool
</span></span></span><span style=margin-top:.5rem class=mr-2 onclick='return window.open("/tags/arduino/","_self"),!1'><span class=flex style=cursor:pointer><span class="rounded-md border border-primary-400 px-1 py-[1px] text-xs font-normal">Arduino
</span></span></span><span style=margin-top:.5rem class=mr-2 onclick='return window.open("/tags/intel/","_self"),!1'><span class=flex style=cursor:pointer><span class="rounded-md border border-primary-400 px-1 py-[1px] text-xs font-normal">Intel
</span></span></span><span style=margin-top:.5rem class=mr-2 onclick='return window.open("/tags/nvidia/","_self"),!1'><span class=flex style=cursor:pointer><span class="rounded-md border border-primary-400 px-1 py-[1px] text-xs font-normal">Nvidia
</span></span></span><span style=margin-top:.5rem class=mr-2 onclick='return window.open("/tags/supermicro/","_self"),!1'><span class=flex style=cursor:pointer><span class="rounded-md border border-primary-400 px-1 py-[1px] text-xs font-normal">Supermicro
</span></span></span><span style=margin-top:.5rem class=mr-2 onclick='return window.open("/tags/aqua-computer/","_self"),!1'><span class=flex style=cursor:pointer><span class="rounded-md border border-primary-400 px-1 py-[1px] text-xs font-normal">Aqua Computer</span></span></span></div></div></div><hr></header><div class="article-content max-w-prose mb-20"><h2 class="relative group">Overview<div id=overview class=anchor></div><span class="absolute top-0 w-6 transition-opacity opacity-0 ltr:-left-6 rtl:-right-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style=text-decoration-line:none!important href=#overview aria-label=Anchor>#</a></span></h2><p>I&rsquo;ve been watercooling my desktop since 2020, and case modding custom cooling solutions since my first modern dGPU in 2012. I enjoy it for the aesthetic, as well as the ability to run my hardware to the redline, despite the lower gains that modern hardware has&mldr; It&rsquo;s still fun to try and see how high you can get on benchmark scoreboards. It was a big initial investment, but with most parts being reusable, the ongoing cost for component upgrades are minimal. Early in 2024, I bought some GPUs to use for dedicated ML tasks in my server rack, and immediately had watercooling on the mind. There weren&rsquo;t particularly strong reasons to do so, but it would lower the power usage a bit, give me some more VRAM overclocking headroom, and give me a bit more core clock stability, as well as the ability to use a cheap two-slot NVLINK connector without suffocating a GPU.</p><hr><div class=panzoom-container><img class="my-0 rounded-md zoomable" src=/posts/watercooling-homelab/images/compressed/Triple_Card_Jank.webp srcset="/posts/watercooling-homelab/images/compressed/Triple_Card_Jank_hu17456463471095554312.webp 330w,
/posts/watercooling-homelab/images/compressed/Triple_Card_Jank_hu3171065713872445154.webp 660w,
/posts/watercooling-homelab/images/compressed/Triple_Card_Jank_hu10619128996450324123.webp 1320w,
/posts/watercooling-homelab/images/compressed/Triple_Card_Jank.webp 2x" alt="My initial setup with 3x 2080 Tis, using m.2 NVMe to PCIe risers in an ASUS prebuilt. Two are connected by NVLINK, which I found to provide a slight performance benefit on the order of ~1-5% in multi-GPU SISR training, which is not worth the typical price for NVLINK bridges from this era. I was lucky to get this ugly, quadro-oriented bridge for just $40." data-src=images/compressed/Triple_Card_Jank.webp></div><figcaption class=caption>My initial setup with 3x 2080 Tis, using m.2 NVMe to PCIe risers in an ASUS prebuilt. Two are connected by NVLINK, which I found to provide a slight performance benefit on the order of ~1-5% in multi-GPU SISR training, which is not worth the typical price for NVLINK bridges from this era. I was lucky to get this ugly, quadro-oriented bridge for just $40.</figcaption><hr><p>I&rsquo;ve never had a leak on my desktop, but with wider temperature swings in my garage, and collectively, a whole lot more expensive hardware that might get damaged than compared to my desktop setup, I was hesitant. The benefits seemed minimal, and I considered it a fun what-if scenario until I upgraded my main server and discovered that the forced-air passive chassis cooling was insufficient for my new CPUs. At that point, I had to make a decision: Get better heatsinks, which would be single-purpose and cost in excess of $100 each, or whole rack watercooling. I chose whole rack watercooling.</p><p>With enough reason for me to go ahead with the project, and the only thing holding me back being a fear of leaks, I had to figure out how to actively monitor for, and preferably prevent such an eventuality. I happened upon a product called <a href=https://shop.aquacomputer.de/Water-Cooling/Leak-testing-prevention/LEAKSHIELD-leak-prevention-system-standalone::4063.html target=_blank>LEAKSHIELD</a>, from Aqua Computer, that advertised itself as doing exactly that.</p><lite-youtube videoid=8UiRv0nDch0 playlabel=8UiRv0nDch0 params></lite-youtube><p>How does it work? You pull a vacuum inside the fluid loop and monitor the loss rate. Water can&rsquo;t get out if air is trying to get in, and if air gets in, the vacuum is reduced. It&rsquo;s so simple, I don&rsquo;t know why it took this long for such a product to enter the PC watercooling space. I could have just bought a LEAKSHIELD and called it a day, but Aqua Computer&rsquo;s software doesn&rsquo;t support Linux, and the specs for the vacuum pump seemed a bit weak for a loop of my scale. But, its functionality was something I was confident I could emulate with more robust off-the-shelf parts. Thus, sufficiently armed with esoteric plumbing knowledge, I took the plunge and started loading up on parts.</p><h2 class="relative group">The Hardware<div id=the-hardware class=anchor></div><span class="absolute top-0 w-6 transition-opacity opacity-0 ltr:-left-6 rtl:-right-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style=text-decoration-line:none!important href=#the-hardware aria-label=Anchor>#</a></span></h2><p>Most of the build uses pretty standard off-the-shelf parts for PC watercooling, but there are a few bits and pieces that most builders won&rsquo;t have seen before, and a custom control system that offers a more relevant experience for a server rack. The control system is based on an Arduino Uno that feeds vital statistics over serial, and features pressure control and monitoring similar to the LEAKSHIELD, with fan control based on a PID algorithm keeping the water temperature at a fixed setpoint above ambient.</p><h3 class="relative group">Off-the-shelf<div id=off-the-shelf class=anchor></div><span class="absolute top-0 w-6 transition-opacity opacity-0 ltr:-left-6 rtl:-right-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style=text-decoration-line:none!important href=#off-the-shelf aria-label=Anchor>#</a></span></h3><h4 class="relative group">General Details<div id=general-details class=anchor></div><span class="absolute top-0 w-6 transition-opacity opacity-0 ltr:-left-6 rtl:-right-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style=text-decoration-line:none!important href=#general-details aria-label=Anchor>#</a></span></h4><p>The centerpiece of the build, which the control unit and pump mount to, is the &ldquo;MOther of all RAdiators&rdquo;, version 3, from Watercool. This is the 360mm version with support for up to 18x 120mm fans, or 8x 180mm fans. It&rsquo;s constructed more in the spirit of a vehicle radiator than a traditional PC radiator, with a less restrictive fin stack and large, round tubes rather than thin rectangular ones. It provides several mounting points for accessories which I was able to utilize to secure it to my server rack in a satisfactorily sturdy fashion. An in-depth teardown on the construction method and material quality of the MO-RA can be found on <a href=https://www.igorslab.de/en/the-big-radiator-material-test-how-much-copper-and-technology-is-in-the-watercool-mo-ra3-360-pro-part-4/ target=_blank>igor&rsquo;sLAB</a>. For fans, I have a collection of old DC-control Corsair Air Series SP120s. They&rsquo;ve all been retired from regular use, because of noise-related aging issues. In fact, one of them failed to turn at all once I had everything wired up, and another had its bearing disintegrate about 8 weeks after putting the thing into service. That being said, they did survive (and continue to survive, in the remaining 16 cases) 24/7 use for anywhere from 4-10 years, at bottom of the barrel pricing, so that&rsquo;s not too bad. I&rsquo;m not exactly pushing the limits of this radiator here, so a few fans breaking down over time isn&rsquo;t the end of the world.</p><hr><div class=panzoom-container><img class="my-0 rounded-md zoomable" src=/posts/watercooling-homelab/images/compressed/MO-RA!.webp srcset="/posts/watercooling-homelab/images/compressed/MO-RA!_hu6856946684884405237.webp 330w,
/posts/watercooling-homelab/images/compressed/MO-RA!_hu15264814207577810858.webp 660w,
/posts/watercooling-homelab/images/compressed/MO-RA!_hu7453820902150892044.webp 1320w,
/posts/watercooling-homelab/images/compressed/MO-RA!.webp 2x" alt="A MO-RA V3 360 PRO PC Watercooling Radiator from [Watercool](https://watercool.de)" data-src=images/compressed/MO-RA!.webp></div><figcaption class=caption>A MO-RA V3 360 PRO PC Watercooling Radiator from <a href=https://watercool.de target=_blank>Watercool</a></figcaption><hr><p>I got a secondhand Corsair XD5 pump/res combo from eBay for about sixty bucks, which is pretty good for a genuine D5-based pump/res combo. It has PWM support which I did wire up, but the flow ended up being rather anemic even at 100%, so I just run it full speed all the time. The flow rate is measured through an <a href=https://shop.aquacomputer.de/Monitoring-and-Controlling/Sensors/Flow-sensor-high-flow-LT-G1-4::3951.html target=_blank>Aqua Computer flow sensor</a>, which is simply a hall-effect tachometer translated to l/h through software. I did not attempt to verify the accuracy of the sensor in my setup. The absolute accuracy is less relevant than simply getting an overall idea of whether or not the measurement is consistent with flow behavior, which it is.</p><hr><div class=panzoom-container><img class="my-0 rounded-md zoomable" src=/posts/watercooling-homelab/images/compressed/Mounting_Detail.webp srcset="/posts/watercooling-homelab/images/compressed/Mounting_Detail_hu3280937936208174423.webp 330w,
/posts/watercooling-homelab/images/compressed/Mounting_Detail_hu1490212990626714303.webp 660w,
/posts/watercooling-homelab/images/compressed/Mounting_Detail_hu11231283172588266533.webp 1320w,
/posts/watercooling-homelab/images/compressed/Mounting_Detail.webp 2x" alt="Simple, cheap aluminum bars and angles mount to the studs on the radiator and into the stud holes on the server rack, and the pump and control box mount onto brackets along with the fans." data-src=images/compressed/Mounting_Detail.webp></div><figcaption class=caption>Simple, cheap aluminum bars and angles mount to the studs on the radiator and into the stud holes on the server rack, and the pump and control box mount onto brackets along with the fans.</figcaption><hr><h4 class="relative group">CPUs<div id=cpus class=anchor></div><span class="absolute top-0 w-6 transition-opacity opacity-0 ltr:-left-6 rtl:-right-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style=text-decoration-line:none!important href=#cpus aria-label=Anchor>#</a></span></h4><p>My thermally problematic server upgrade was to dual Xeon Gold 6154s, which are Skylake-SP architecture. This specific SKU is pretty beefy, with 18 cores at sustained all-core speeds of 3.7GHz SSE / 3.3GHz AVX2 / 2.7GHz AVX512, and a TDP of 200 watts. The rated tjmax was 105C, and with the chassis cooling, they readily met that and started throttling under all-core loads, with idle as high as 60-70*C. I previously had Xeon E5-2697 v2s, which had TDPs of 130w. They got toasty, but never throttled. I&rsquo;m not sure if the chassis had any easy fan upgrades available that might have made a difference, and I certainly could have moved to 4u-compatible tower coolers rather than forced air, but since I wanted to watercool the GPUs anyway, adding the CPUs as well would be minimal cost/effort, with more future compatibility for the waterblocks compared to a specialized LGA3647 tower cooler.</p><hr><div class=panzoom-container><img class="my-0 rounded-md zoomable" src=/posts/watercooling-homelab/images/compressed/Coldplate.webp srcset="/posts/watercooling-homelab/images/compressed/Coldplate_hu13174017899928575525.webp 330w,
/posts/watercooling-homelab/images/compressed/Coldplate_hu3290250784134948227.webp 660w,
/posts/watercooling-homelab/images/compressed/Coldplate_hu16258961307282205929.webp 1320w,
/posts/watercooling-homelab/images/compressed/Coldplate.webp 2x" alt="Alphacool Eisblock XPX Pro coldplate. Image credit & copyright - [igor'sLAB](https://www.igorslab.de/en/ryzen-threadripper-2990-wx-with-500-w-alphacool-iceblock-xpx-aurora-pro-plexi-digital-rgb-in-test/)" data-src=images/compressed/Coldplate.webp></div><figcaption class=caption>Alphacool Eisblock XPX Pro coldplate Image credit & copyright - <a href=https://www.igorslab.de/en/ryzen-threadripper-2990-wx-with-500-w-alphacool-iceblock-xpx-aurora-pro-plexi-digital-rgb-in-test/ target=_blank>igor&rsquo;sLAB</a></figcaption><hr><p>The CPU waterblocks are Alphacool Eisblock XPX Pro Aurora Light models, which are significantly cheaper than the XPX Aurora Pro not-light version. They appear to be entirely identical, functionally&mldr; I&rsquo;m not sure if there any actual performance benefits offered by the not-light version. It&rsquo;s a relatively obscure block family without many thorough reviews, which makes sense, given this block is designed for full coverage on Xeons/Threadrippers. The coldplate appears to be <a href=https://en.wikipedia.org/wiki/Skiving_%28metalworking%29#Heat_sinks target=_blank>skived</a>, which is uncommon in this price bracket for a discrete block, and the fins are incredibly short and dense. In smaller desktop loops, I&rsquo;ve seen this block criticized for having overly restrictive flow, but when you have four blocks + quick disconnects, &ldquo;good&rdquo; flow is relative. At the power limit of 200w, the maximum core temperature delta relative to the water temperature is 25*C, with a ~1-2*C average delta between the two serially-connected sockets at a flow rate of ~130L/h, and that&rsquo;s more than sufficient.</p><hr><div class=panzoom-container><img class="my-0 rounded-md zoomable" src=/posts/watercooling-homelab/images/compressed/Dual_Blocks_Zoom.webp srcset="/posts/watercooling-homelab/images/compressed/Dual_Blocks_Zoom_hu1716452709661764841.webp 330w,
/posts/watercooling-homelab/images/compressed/Dual_Blocks_Zoom_hu478575305574189147.webp 660w,
/posts/watercooling-homelab/images/compressed/Dual_Blocks_Zoom_hu1233157730881247055.webp 1320w,
/posts/watercooling-homelab/images/compressed/Dual_Blocks_Zoom.webp 2x" alt="Interior view of the Supermicro CSE-846 chassis showcasing the installed waterblocks and other components." data-src=images/compressed/Dual_Blocks_Zoom.webp></div><figcaption class=caption>Interior view of the Supermicro CSE-846 chassis showcasing the installed waterblocks and other components.</figcaption><hr><h4 class="relative group">GPUs<div id=gpus class=anchor></div><span class="absolute top-0 w-6 transition-opacity opacity-0 ltr:-left-6 rtl:-right-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style=text-decoration-line:none!important href=#gpus aria-label=Anchor>#</a></span></h4><p>The GPU blocks are Phanteks 2080 Ti Founder&rsquo;s Edition blocks. Nothing special, they&rsquo;re just the cheapest matching ones I could find in 2024 that looked like they&rsquo;d fit these almost-reference-but-not-quite OEM cards. They&rsquo;re generic OEM models that would have gone in prebuilts. The most interesting thing about these cards, is that they&rsquo;ve been modded to have 22GB of VRAM. There&rsquo;s a <a href=https://2080ti22g.com/ title="#not an ad, but it could be 🪝☝️😜" target=_blank>dedicated supplier</a> still offering them, and it&rsquo;s by far the best $/GB value for VRAM in modern NVIDIA GPUs.<sup id=fnref:1><a href=#fn:1 class=footnote-ref role=doc-noteref>1</a></sup> Whether or not this is a better value overall than, say, a 3090 (Ti) depends on your usecase. Performance improvements in ML tasks between the 2080 Ti and 3090 (Ti) range from as little as ~20% to as much as ~100% depending on how memory bandwidth constrained your workload is. With secondhand 3090 (Ti)s still going for minimum $700 on the used market in the US, I found the alternative 2080 Ti option to be more alluring for my usecase, which is primarily single image super-resolution. More VRAM is desirable to increase the size of tiles for inference, and to increase the batch size during training. Training speed scales almost linearly, and inference speed scales linearly, per GPU. So, for my usecase, where I&rsquo;m not really limited by the performance of a single GPU, the 2080 Ti mod route offers better overall value both for VRAM and combined core performance compared to 3090 (Ti)s. The idea of having a modded GPU in itself was also appealing and definitely part of why I made that decision. Pulling up a hardware monitor and seeing a 2080 Ti with 22GB of VRAM feels a little bit naughty, and I like that.</p><hr><div class=panzoom-container><img class="my-0 rounded-md zoomable" src=/posts/watercooling-homelab/images/compressed/GPUs_Installed.webp srcset="/posts/watercooling-homelab/images/compressed/GPUs_Installed_hu4193758725753816349.webp 330w,
/posts/watercooling-homelab/images/compressed/GPUs_Installed_hu1923429014339473058.webp 660w,
/posts/watercooling-homelab/images/compressed/GPUs_Installed_hu1860541658624349795.webp 1320w,
/posts/watercooling-homelab/images/compressed/GPUs_Installed.webp 2x" alt="The blocks installed in an ASUS prebuilt gaming tower." data-src=images/compressed/GPUs_Installed.webp></div><figcaption class=caption>The blocks installed in an ASUS prebuilt gaming tower.</figcaption><hr><p>I did initially buy three of them, as pictured at the beginning of this post, but one of them failed just after the 30 day warranty period listed on their website. Despite that, they were kind enough to offer a full refund if I covered return shipping, and were very communicative and responded in &lt;24 hours every time I sent them any kind of message/inquiry.</p><p>The biggest benefit that watercooling theoretically brings to modern video cards is a prolonged lifespan. Not due to lower core temperatures,<sup id=fnref:2><a href=#fn:2 class=footnote-ref role=doc-noteref>2</a></sup> in an absolute sense, but due to the reduced stress from thermal cycles. Mismatches in the rate of thermal expansion between the die and the substrate will eventually cause their bond to break, and this happens faster the larger your die is, and the more extreme the temperature differences are. Today&rsquo;s GPU dies are huge, and it&rsquo;s hard to say how many failures are attributable to this factor alone, but it is certain to be more of a risk than it has been in the past. I&rsquo;d rather buy a used mining GPU than a used gaming GPU any day, because it has likely been kept at roughly the same temperature for most of its life, as opposed to experiencing wide periodic swings.</p><hr><div id=gallery-478512900c1850c6c722f5e31b1ee9f5 class=gallery><div class="panzoom-container grid-w25"><img class="my-0 rounded-md zoomable" src=/posts/watercooling-homelab/images/compressed/Block_Mod_Detail_A.webp srcset="/posts/watercooling-homelab/images/compressed/Block_Mod_Detail_A_hu6560107379008232574.webp 330w,
/posts/watercooling-homelab/images/compressed/Block_Mod_Detail_A_hu1954630252676748703.webp 660w,
/posts/watercooling-homelab/images/compressed/Block_Mod_Detail_A_hu7088133987632116537.webp 1320w,
/posts/watercooling-homelab/images/compressed/Block_Mod_Detail_A.webp 2x" alt="Trimmed area for the capacitor." data-src=images/compressed/Block_Mod_Detail_A.webp></div><div class="panzoom-container grid-w25"><img class="my-0 rounded-md zoomable" src=/posts/watercooling-homelab/images/compressed/Block_Mod_Detail_B.webp srcset="/posts/watercooling-homelab/images/compressed/Block_Mod_Detail_B_hu2333850158973680984.webp 330w,
/posts/watercooling-homelab/images/compressed/Block_Mod_Detail_B_hu5952309977048855875.webp 660w,
/posts/watercooling-homelab/images/compressed/Block_Mod_Detail_B_hu12155478021797826213.webp 1320w,
/posts/watercooling-homelab/images/compressed/Block_Mod_Detail_B.webp 2x" alt="An area of the block cut out to make room for the power plugs." data-src=images/compressed/Block_Mod_Detail_B.webp></div><div class="panzoom-container grid-w50"><img class="my-0 rounded-md zoomable" src=/posts/watercooling-homelab/images/compressed/Tall_Capacitor.webp srcset="/posts/watercooling-homelab/images/compressed/Tall_Capacitor_hu3985119007043451200.webp 330w,
/posts/watercooling-homelab/images/compressed/Tall_Capacitor_hu11882410216080315790.webp 660w,
/posts/watercooling-homelab/images/compressed/Tall_Capacitor_hu1042940534242500862.webp 1320w,
/posts/watercooling-homelab/images/compressed/Tall_Capacitor.webp 2x" alt="Showcasing the capacitor fitting into the trimmed area." data-src=images/compressed/Tall_Capacitor.webp></div><div class="panzoom-container grid-w50"><img class="my-0 rounded-md zoomable" src=/posts/watercooling-homelab/images/compressed/Different_Plugs.webp srcset="/posts/watercooling-homelab/images/compressed/Different_Plugs_hu17890886748642996535.webp 330w,
/posts/watercooling-homelab/images/compressed/Different_Plugs_hu2378229743182206831.webp 660w,
/posts/watercooling-homelab/images/compressed/Different_Plugs_hu3502703824848779819.webp 1320w,
/posts/watercooling-homelab/images/compressed/Different_Plugs.webp 2x" alt="Showcasing the plugs fitting into the cutout area." data-src=images/compressed/Different_Plugs.webp></div></div><figcaption class=caption>The GPU blocks required a <em>moderate amount of light massaging</em> to properly fit on these OEM model cards. The power plugs are in a different position and a singular capacitor on these models is slightly taller than on the actual Founder&rsquo;s Edition reference card, but they&rsquo;re otherwise close enough to identical.</figcaption><hr><p>That&rsquo;s not to say that there are no benefits from lowering the operating temperature. As an absolute value, within manufacturer limits, it affects boost clocks, and <a href=https://en.wikipedia.org/wiki/Leakage_%28electronics%29#In_semiconductors target=_blank>leakage current</a>. A cooler chip will use less power to run at the same clock speed compared to a hotter chip due to reduced leakage current, making them measurably more energy efficient per clock cycle the colder they run. In my case, with the fan on max, while not thermal throttling, these GPUs would bounce off the power limit of 280w while attempting to hit a core clock of 1800MHz. Under water, at a reported core temperature of ~30*C, the reported board power draw is only ~220w at 1800MHz<sup id=fnref:3><a href=#fn:3 class=footnote-ref role=doc-noteref>3</a></sup> core clock for the same workload. The type of fan typically found in these coolers is rated anywhere from 15-30w on its own, so a reduction of at least 30w can likely be attributed to a lower leakage current.</p><h3 class="relative group">DIY Time<div id=diy-time class=anchor></div><span class="absolute top-0 w-6 transition-opacity opacity-0 ltr:-left-6 rtl:-right-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style=text-decoration-line:none!important href=#diy-time aria-label=Anchor>#</a></span></h3><p>In no particular order, here is a list of the major components involved in the control system.</p><ul><li>Generic metal box, formerly from a PBX system.</li><li>Arduino Uno clone, unknown brand</li><li>60mm Corsair fan</li><li>RS232 TTL shifter</li><li>Aesthetic retro power switch</li><li>12v DC vacuum pump</li><li>U.S. Solid 12V NC Solenoid</li><li>12v relay modules</li><li>HX711 ADC</li><li>MD-PS002 Absolute Pressure Sensors</li><li>L298-like PWM motor driver</li><li>Apple White iMac PSU</li><li>Adafruit Arduino Uno Proto Shield</li><li>DS18B20 temperature probes</li></ul><hr><div class=panzoom-container><img class="my-0 rounded-md zoomable" src=/posts/watercooling-homelab/images/compressed/Test_Fit.webp srcset="/posts/watercooling-homelab/images/compressed/Test_Fit_hu6466058294895506811.webp 330w,
/posts/watercooling-homelab/images/compressed/Test_Fit_hu4196319411009163019.webp 660w,
/posts/watercooling-homelab/images/compressed/Test_Fit_hu2884804705215397458.webp 1320w,
/posts/watercooling-homelab/images/compressed/Test_Fit.webp 2x" alt="Fit check for all the major components." data-src=images/compressed/Test_Fit.webp></div><figcaption class=caption>Fit check for all the major components.</figcaption><hr><p>I didn&rsquo;t take excruciatingly detailed pictures of every single step of the assembly/prototyping process. For the most part, I was just plugging pre-made components together. The most interesting production notes include the pressure sensor and the power supply.</p><h4 class="relative group">Putting New Life into an iMac PSU<div id=putting-new-life-into-an-imac-psu class=anchor></div><span class="absolute top-0 w-6 transition-opacity opacity-0 ltr:-left-6 rtl:-right-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style=text-decoration-line:none!important href=#putting-new-life-into-an-imac-psu aria-label=Anchor>#</a></span></h4><p>The power supply I used is from a first-gen Intel White iMac, which is visually very similar to the G5. It was one of the earliest things that I installed Linux on, and I used it as a seedbox for a bit, but eventually took it apart and saved some of the more interesting stuff.</p><p>All the credit goes to the user <em>ersterhernd</em>, from <a href=https://www.tonymacx86.com/threads/imac-isight-model-power-supply-unleashed.150793/ target=_blank>this thread</a> on the <a href=https://tonymacx86.com target=_blank>tonymacx86.com</a> forum for figuring out the pinout of this PSU, which is almost entirely identical to the one that was in my unit, apart from the power rating on mine being 200w. There are two banks of pins, half of which are always on, half of which are toggleable. Each bank has 12v, 5v, and 3.3v. I didn&rsquo;t end up using 3.3v for anything other than the power switch. I have no idea what the energy efficiency of this unit is, obviously it doesn&rsquo;t have an 80+ certification&mldr; But I&rsquo;m assuming that Apple would make it at least halfway decent. Hopefully more efficient than a random 12v power brick with additional converters, I&rsquo;d hope.</p><hr><div class=panzoom-container><img class="my-0 rounded-md zoomable" src=/posts/watercooling-homelab/schematic_minify.svg alt="My schematic for the control unit. It was the first time I've ever used KiCad, and the first time I've ever made a schematic like this at all. I hope it's relatively legible." data-src=schematic_minify.svg></div><figcaption class=caption>My schematic for the control unit. It was the first time I&rsquo;ve ever used KiCad, and the first time I&rsquo;ve ever made a schematic like this at all. I hope it&rsquo;s relatively legible.</figcaption><hr><p>As you can see in the schematic above, the always-on 3.3v pin is connected to SYS_POWERUP through a relay board. The relay input is pulled low by a single pole switch, which turns the relay on, which connects ground to SYS_POWERUP, engaging the other rail of the power supply. This is kind of a convoluted solution to not having a double-pole switch&mldr; But I didn&rsquo;t have a double-pole switch, so that&rsquo;s what I did.</p><h4 class="relative group">Measuring vacuum<div id=measuring-vacuum class=anchor></div><span class="absolute top-0 w-6 transition-opacity opacity-0 ltr:-left-6 rtl:-right-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style=text-decoration-line:none!important href=#measuring-vacuum aria-label=Anchor>#</a></span></h4><p>The leak-resisting aspect all hinges on monitoring the pressure of the loop&mldr; Or potentially running a vacuum pump constantly, but that&rsquo;s stupid. For some reason, I had a really hard time finding a vacuum pressur sensor. There are plenty of physical, analogue vacuum gauges available, but as far as an electronic sensor&mldr; I just couldn&rsquo;t find any located in the US, at a reasonable price. There were a few hobbyist grade differential sensors, but I wanted to be able to measure down to an almost complete vacuum, and they didn&rsquo;t have the range. Maybe I had the wrong search terms, but I just wasn&rsquo;t finding anything. Eventually I found an unpackaged sensor with obscure, not entirely legible datasheets that claimed to have an acceptable pressure range for my application. The <a href=https://electronperdido.com/wp-content/uploads/2021/12/MD_PS002-Datasheet.zh-CN.en_.pdf target=_blank>MD-PS002</a> is what I settled on, available on Amazon in the US in a 2-pack for $8. It&rsquo;s a tiny little thing, and it took two attempts to successfully create a sensor package that didn&rsquo;t leak.</p><hr><div id=gallery-72e13e2551c0e142bf6afee01ef5c126 class=gallery><div class="panzoom-container grid-w50"><img class="my-0 rounded-md zoomable" src=/posts/watercooling-homelab/images/compressed/plug_detail_top.webp srcset="/posts/watercooling-homelab/images/compressed/plug_detail_top_hu7787027711586187933.webp 330w,
/posts/watercooling-homelab/images/compressed/plug_detail_top_hu16358936284331608818.webp 660w,
/posts/watercooling-homelab/images/compressed/plug_detail_top_hu13627305942680874943.webp 1320w,
/posts/watercooling-homelab/images/compressed/plug_detail_top.webp 2x" alt="Top view of the sensor JB Welded into the drilled out plug." data-src=images/compressed/plug_detail_top.webp></div><div class="panzoom-container grid-w50"><img class="my-0 rounded-md zoomable" src=/posts/watercooling-homelab/images/compressed/plug_detail_bottom.webp srcset="/posts/watercooling-homelab/images/compressed/plug_detail_bottom_hu3645814594706023691.webp 330w,
/posts/watercooling-homelab/images/compressed/plug_detail_bottom_hu12744847709972265499.webp 660w,
/posts/watercooling-homelab/images/compressed/plug_detail_bottom_hu3698649971536149681.webp 1320w,
/posts/watercooling-homelab/images/compressed/plug_detail_bottom.webp 2x" alt="Bottom view of the sensor JB Welded into the drilled out plug." data-src=images/compressed/plug_detail_bottom.webp></div><div class="panzoom-container grid-w100"><img class="my-0 rounded-md zoomable" src=/posts/watercooling-homelab/images/compressed/goopy_installed.webp srcset="/posts/watercooling-homelab/images/compressed/goopy_installed_hu13878470882578617902.webp 330w,
/posts/watercooling-homelab/images/compressed/goopy_installed_hu7775517930751145146.webp 660w,
/posts/watercooling-homelab/images/compressed/goopy_installed_hu8582164235252167480.webp 1320w,
/posts/watercooling-homelab/images/compressed/goopy_installed.webp 2x" alt="Sensor package with additional JB Weld installed into the vacuum tank." data-src=images/compressed/goopy_installed.webp></div></div><figcaption class=caption>Sensor package details, installed and all gooped up.</figcaption><hr><p>I drilled a hole in a G1/4" plug, just slightly bigger than the metal ring on the sensor, coated that ring with J-B Weld, and inserted it, letting it cure before grinding away the exterior of the top of the plug and building up more J-B weld to add some strain relief for the wires as well as edge-to-edge sealing. The current vacuum loss rate, after running the system for a few months, allowing the loop to very thoroughly de-gas, is now less than 50mbar per day at -500 to -600mbar. I was slightly worried about the lifetime of the pump, given it&rsquo;s a cheap thing from Amazon, but given it only has to run for about a second every 2-3 days, I imagine that won&rsquo;t be an issue.</p><p>Here&rsquo;s a quick video showing the system not leaking!</p><lite-youtube videoid=wCmtzoBKFZg playlabel=wCmtzoBKFZg params></lite-youtube><p>This pressure sensor is a wheatstone bridge, which works the same way as load cells for digital scales. The resistance changes are very, very low, thus the signal must be amplified before being fed into an ADC. You could use an op-amp, and feed that signal into an analog input on the Arduino, but I felt more comfortable using the HX711, a two-channel ADC with integrated amplifier designed to be used with wheatstone bridge load cells. Here&rsquo;s a code snippet showing how I converted the raw analog measurement to mbar.</p><div class=highlight><pre tabindex=0 class=chroma><code class=language-c++ data-lang=c++><span class=line><span class=cl><span class=kt>float</span> <span class=nf>pressure_raw_to_mbar</span><span class=p>(</span><span class=kt>int32_t</span> <span class=n>pressure_raw</span><span class=p>)</span> <span class=p>{</span>
</span></span><span class=line><span class=cl> <span class=k>return</span> <span class=p>(</span><span class=n>pressure_raw</span> <span class=o>-</span> <span class=mi>390000</span><span class=p>)</span> <span class=o>*</span> <span class=p>(</span><span class=mf>1700.0</span> <span class=o>/</span> <span class=p>(</span><span class=mi>5600000</span> <span class=o>-</span> <span class=mi>390000</span><span class=p>))</span> <span class=o>-</span> <span class=mi>700</span><span class=p>;</span>
</span></span><span class=line><span class=cl><span class=p>}</span>
</span></span></code></pre></div><p>I calibrated it manually, comparing it to an analogue gauge. It&rsquo;s calibrated to a zero point at atmospheric pressure in my locale, and from -700mbar to +1000mbar. I figured out that, when setting the HX711 to a gain of 64 with the Adafruit HX711 library, a change of 100mbar is a change in the ADC measurement by 30k, highly consistent across the entire pressure range that I tested. I can&rsquo;t be 100% sure how accurate the analogue gauge is, but 100% accuracy doesn&rsquo;t really matter for this application. All I really need to know is the fact that an adequate vacuum is present, and a general idea of the leak rate, which is a requirement that this setup meets.</p><h4 class="relative group">Other stuff<div id=other-stuff class=anchor></div><span class="absolute top-0 w-6 transition-opacity opacity-0 ltr:-left-6 rtl:-right-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style=text-decoration-line:none!important href=#other-stuff aria-label=Anchor>#</a></span></h4><p>I got a beefy PWM motor driver with L298 logic, claiming a continuous current of 7 amps per channel, which nicely fit my requirements. 120mm PC fans are typically 0.2-0.3 amps, and mine in particular are 0.25. For 18 fans, it should be approximately 4.5 amps at 100% speed. It&rsquo;s a bit oversized, and I&rsquo;m only using one channel, but it leaves me the option in the future to use larger, generic radiator fans that have more demanding power requirements. I&rsquo;m already down two from eighteen, eventually enough of them are going to fail that I&rsquo;ll have to find another solution.</p><hr><div id=gallery-72362228f901955b607c691a5e4e1459 class=gallery><div class="panzoom-container grid-w45"><img class="my-0 rounded-md zoomable" src=/posts/watercooling-homelab/images/compressed/pump_greeble.webp srcset="/posts/watercooling-homelab/images/compressed/pump_greeble_hu1471177938307566585.webp 330w,
/posts/watercooling-homelab/images/compressed/pump_greeble_hu4319233910559987333.webp 660w,
/posts/watercooling-homelab/images/compressed/pump_greeble_hu1040829716072557719.webp 1320w,
/posts/watercooling-homelab/images/compressed/pump_greeble.webp 2x" alt="Top view of the sensor JB Welded into the drilled out plug." data-src=images/compressed/pump_greeble.webp></div><div class="panzoom-container grid-w55"><img class="my-0 rounded-md zoomable" src=/posts/watercooling-homelab/images/compressed/complete_assembly.webp srcset="/posts/watercooling-homelab/images/compressed/complete_assembly_hu13996029485759007548.webp 330w,
/posts/watercooling-homelab/images/compressed/complete_assembly_hu402138244258743400.webp 660w,
/posts/watercooling-homelab/images/compressed/complete_assembly_hu17913719379729244320.webp 1320w,
/posts/watercooling-homelab/images/compressed/complete_assembly.webp 2x" alt="Bottom view of the sensor JB Welded into the drilled out plug." data-src=images/compressed/complete_assembly.webp></div><div class="panzoom-container grid-w45"><img class="my-0 rounded-md zoomable" src=/posts/watercooling-homelab/images/compressed/solenoid_diode.webp srcset="/posts/watercooling-homelab/images/compressed/solenoid_diode_hu13681423255705621406.webp 330w,
/posts/watercooling-homelab/images/compressed/solenoid_diode_hu3904316093352162951.webp 660w,
/posts/watercooling-homelab/images/compressed/solenoid_diode_hu14196543121366269339.webp 1320w,
/posts/watercooling-homelab/images/compressed/solenoid_diode.webp 2x" alt="Sensor package with additional JB Weld installed into the vacuum tank." data-src=images/compressed/solenoid_diode.webp></div></div><figcaption class=caption>Required additions to the solenoid, pump motor, and the complete assembly without cover.</figcaption><hr><p>In my initial tests, I found that operating the pump and solenoid would cause the Arduino to reset, seemingly at random, or cause other undefined behavior. Since they were not electrically isolated on a second power supply, that makes sense. They were backfeeding energy and causing a notable amount of general interference during operation, to the point that the LEDs on the inactive relay modules would dimly illuminate when the motor was in operation, and very visibly illuminate whenever the motor or the solenoid deactivated. I had to add flyback diodes, and, for peace of mind, I added ceramic filtering capacitors to the pump as well. Those additions completely eliminated the issue. Below is a video demonstrating the bad behavior.</p><lite-youtube videoid=E-Ngy0T2RyM playlabel=E-Ngy0T2RyM params></lite-youtube><p>I did a similar plug-drilling setup for the water temperature sensor with a generic Dallas temperature probe. The air probe was taped to the exterior of the box, in the path of the incoming air. All that remained was to solder up a sort of bus bar for the radiator fan connectors, get the temperature probes and pull-up resistors wired into the proto board, and hook everything up to the Arduino, then write the software to tie it all together.</p><h2 class="relative group">The Software<div id=the-software class=anchor></div><span class="absolute top-0 w-6 transition-opacity opacity-0 ltr:-left-6 rtl:-right-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style=text-decoration-line:none!important href=#the-software aria-label=Anchor>#</a></span></h2><a id=gitea-5e07c81c36523e950f8989feb10b664d target=_blank href=https://git.neet.works/rawhide_k/server-watercooling-controller class=cursor-pointer><div class="w-full md:w-auto pt-3 p-5 border border-neutral-200 dark:border-neutral-700 border rounded-md shadow-2xl"><div class="flex items-center"><span class="text-2xl text-neutral-800 dark:text-neutral" style=margin-right:10px><span class="relative block icon"><svg viewBox="5.67 143.05 628.65 387.55"><path fill="currentcolor" d="M115.912 143.075c-6.462.0-13.762.525-22.012 2.325-8.7 1.8-33.5 7.4-53.8 26.9C-4.9 212.4 6.6 276.2 8 285.8c1.7 11.7 6.9 44.2 31.7 72.5 45.8 56.1 144.4 54.8 144.4 54.8s12.1 28.9 30.6 55.5c25 33.1 50.7 58.9 75.7 62 63 0 188.9-.1 188.9-.1s12 .1 28.3-10.3c14-8.5 26.5-23.4 26.5-23.4S547 483 565 451.5c5.5-9.7 10.1-19.1 14.1-28 0 0 55.2-117.1 55.2-231.1-1.1-34.5-9.6-40.6-11.6-42.6-4.1-4.1-9.6-4-9.6-4s-117.2 6.6-177.9 8c-13.267.3-26.433.597-39.5.697l.1 117.002s57.4 24.202 83.1 40.102c3.7 2.3 10.2 6.798 12.9 14.398 2.1 6.1 2 13.101-1 19.301l-61 126.9c-6.2 12.7-21.4 18.1-33.9 12l-126.9-61c-12.5-6-17.9-21.2-11.8-33.8l61-126.9c6-12.5 21.2-17.9 33.8-11.8a5290.322 5290.322.0 0027 12.954c0-36.449-.1-109.053-.1-109.053-29 .4-89.2-2.201-89.2-2.201s-141.4-7.1-156.8-8.5c-4.9-.3-10.525-.825-16.988-.825zm12.188 48.026s7.1 59.399 15.7 94.199c7.2 29.2 24.8 77.7 24.8 77.7s-26.1-3.1-43-9.1c-25.9-8.5-36.9-18.7-36.9-18.7S69.6 321.8 60 295.4c-16.5-44.2-1.4-71.2-1.4-71.2s8.4-22.5 38.5-30c13.8-3.7 31-3.1 31-3.1z"/><path fill="currentcolor" d="M326.8 380.1c-8.2.1-15.4 5.8-17.3 13.8-1.9 8 2 16.3 9.1 20 7.7 4 17.5 1.8 22.7-5.4 5.1-7.1 4.3-16.9-1.8-23.1l24-49.1c1.5.1 3.7.2 6.2-.5 4.1-.9 7.1-3.6 7.1-3.6 4.2 1.8 8.6 3.8 13.2 6.1 4.8 2.4 9.3 4.9 13.4 7.3.9.5 1.8 1.1 2.8 1.9 1.6 1.3 3.4 3.1 4.7 5.5 1.9 5.5-1.9 14.9-1.9 14.9-2.3 7.6-18.4 40.6-18.4 40.6-8.1-.2-15.3 5-17.7 12.5-2.6 8.1 1.1 17.3 8.9 21.3 7.8 4 17.4 1.7 22.5-5.3 5-6.8 4.6-16.3-1.1-22.6 1.9-3.7 3.7-7.4 5.6-11.3 5-10.4 13.5-30.4 13.5-30.4.9-1.7 5.7-10.3 2.7-21.3-2.5-11.4-12.6-16.7-12.6-16.7-12.2-7.9-29.2-15.2-29.2-15.2s0-4.1-1.1-7.1c-1.1-3.1-2.8-5.1-3.9-6.3 4.7-9.7 9.4-19.3 14.1-29-4.1-2-8.1-4-12.2-6.1-4.8 9.8-9.7 19.7-14.5 29.5-6.7-.1-12.9 3.5-16.1 9.4-3.4 6.3-2.7 14.1 1.9 19.8l-24.6 50.4z"/></svg></span></span><div id=gitea-5e07c81c36523e950f8989feb10b664d-full_name class="m-0 font-bold text-xl text-neutral-800 decoration-primary-500 hover:underline hover:underline-offset-2 dark:text-neutral">rawhide_k/server-watercooling-controller</div></div><p id=gitea-5e07c81c36523e950f8989feb10b664d-description class="m-0 mt-2 text-md text-neutral-800 dark:text-neutral"></p><div class="m-0 mt-2 flex items-center"><span class="mr-1 inline-block h-3 w-3 rounded-full" style=background-color:#f34b7d></span><div class="m-0 mr-5 text-md text-neutral-800 dark:text-neutral">C++</div><span class="text-md mr-1 text-neutral-800 dark:text-neutral"><span class="relative block icon"><svg viewBox="0 0 576 512"><path fill="currentcolor" d="M287.9.0C297.1.0 305.5 5.25 309.5 13.52L378.1 154.8l153.3 22.7C540.4 178.8 547.8 185.1 550.7 193.7 553.5 202.4 551.2 211.9 544.8 218.2L433.6 328.4l26.3 155.5C461.4 492.9 457.7 502.1 450.2 507.4 442.8 512.7 432.1 513.4 424.9 509.1L287.9 435.9 150.1 509.1C142.9 513.4 133.1 512.7 125.6 507.4 118.2 502.1 114.5 492.9 115.1 483.9l27.1-155.5L31.11 218.2C24.65 211.9 22.36 202.4 25.2 193.7 28.03 185.1 35.5 178.8 44.49 177.5L197.7 154.8 266.3 13.52C270.4 5.249 278.7.0 287.9.0zm0 78.95L235.4 187.2C231.9 194.3 225.1 199.3 217.3 200.5L98.98 217.9 184.9 303C190.4 308.5 192.9 316.4 191.6 324.1L171.4 443.7l105.2-56.2C283.7 383.7 292.2 383.7 299.2 387.5l105.2 56.2-20.2-119.6C382.9 316.4 385.5 308.5 391 303l85.9-85.1-118.3-17.4C350.7 199.3 343.9 194.3 340.5 187.2L287.9 78.95z"/></svg></span></span><div id=gitea-5e07c81c36523e950f8989feb10b664d-stars_count class="m-0 mr-5 text-md text-neutral-800 dark:text-neutral">0</div><span class="text-md mr-1 text-neutral-800 dark:text-neutral"><span class="relative block icon"><svg viewBox="0 0 448 512"><path fill="currentcolor" d="M80 104c13.3.0 24-10.7 24-24S93.3 56 80 56 56 66.7 56 80s10.7 24 24 24zm80-24c0 32.8-19.7 61-48 73.3V192c0 17.7 14.3 32 32 32H304c17.7.0 32-14.3 32-32V153.3C307.7 141 288 112.8 288 80c0-44.2 35.8-80 80-80s80 35.8 80 80c0 32.8-19.7 61-48 73.3V192c0 53-43 96-96 96H256v70.7c28.3 12.3 48 40.5 48 73.3.0 44.2-35.8 80-80 80s-80-35.8-80-80c0-32.8 19.7-61 48-73.3V288H144c-53 0-96-43-96-96V153.3C19.7 141 0 112.8.0 80 0 35.8 35.8.0 80 0s80 35.8 80 80zm208 24c13.3.0 24-10.7 24-24s-10.7-24-24-24-24 10.7-24 24 10.7 24 24 24zM248 432c0-13.3-10.7-24-24-24s-24 10.7-24 24 10.7 24 24 24 24-10.7 24-24z"/></svg></span></span><div id=gitea-5e07c81c36523e950f8989feb10b664d-forks_count class="m-0 mr-5 text-md text-neutral-800 dark:text-neutral">0</div></div></div><script>fetch("https://git.neet.works/api/v1/repos/rawhide_k/server-watercooling-controller",{headers:new Headers({"User-agent":"Mozilla/4.0 Custom User Agent"})}).then(e=>e.json()).then(e=>{document.getElementById("gitea-5e07c81c36523e950f8989feb10b664d-full_name").innerHTML=e.full_name,document.getElementById("gitea-5e07c81c36523e950f8989feb10b664d-description").innerHTML=e.description,document.getElementById("gitea-5e07c81c36523e950f8989feb10b664d-stars_count").innerHTML=e.stars_count,document.getElementById("gitea-5e07c81c36523e950f8989feb10b664d-forks_count").innerHTML=e.forks_count}).catch(e=>console.error(e))</script></a><p>The Arduino operates independently, without a server. The fan speed and loop pressure are managed autonomously. The serial connection is only used to report vital stats, for later integration into more connected monitoring systems. Ultimately, there are only two actions it can take: Change the fan speed, and turn on the vacuum pump. Neither of these require any external knowledge. It doesn&rsquo;t need any information about the connected hardware to function correctly. All it&rsquo;s concerned with is keeping the water temperature at a certain delta above ambient temperature, and keeping the loop pressure within a certain range. Ultimately, very simple.</p><p>Once per second, the temperature sensors are sampled, the PID loop for the fans runs with the new temperature data points, and the fan speed, temperature data, vacuum pressure, and pump/flow measurements are sent over serial with the help of the ArduinoJSON library. I settled on a target water delta of 4*C relative to ambient, with a chosen min/max temperature range where the fans turn off or pin themselves to 100% completely. The 4*C delta is rather arbitrary. It&rsquo;s approximately the delta that exists when the systems are on, but idle, and the fans are at their minimum speed. That delta can be maintained during 100% CPU load, and during medium-heavy GPU loads, but not both combined. It still stays well under a 10*C delta in that case, though, so I can&rsquo;t complain.</p><p>There are also hard stops to turn the fans off if the water temperature hits 5*C, and pin them to max if it hits 40*C. I&rsquo;m not sure how realistic either of those figures are, but it&rsquo;s better to be safe than frozen up and/or exploded.</p><div class=highlight><pre tabindex=0 class=chroma><code class=language-c++ data-lang=c++><span class=line><span class=cl><span class=kt>int</span> <span class=nf>fan_PID</span><span class=p>(</span><span class=kt>float</span><span class=o>*</span> <span class=n>air_temp</span><span class=p>,</span> <span class=kt>float</span><span class=o>*</span> <span class=n>water_temp</span><span class=p>,</span> <span class=kt>uint32_t</span><span class=o>*</span> <span class=n>cur_loop_timestamp</span><span class=p>)</span> <span class=p>{</span>
</span></span><span class=line><span class=cl> <span class=k>static</span> <span class=k>const</span> <span class=kt>float</span> <span class=n>kp</span> <span class=o>=</span> <span class=mf>120.0</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=k>static</span> <span class=k>const</span> <span class=kt>float</span> <span class=n>ki</span> <span class=o>=</span> <span class=mf>0.16</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=k>static</span> <span class=k>const</span> <span class=kt>float</span> <span class=n>kd</span> <span class=o>=</span> <span class=mf>4.0</span><span class=p>;</span>
</span></span><span class=line><span class=cl>
</span></span><span class=line><span class=cl> <span class=k>static</span> <span class=kt>float</span> <span class=n>integral</span> <span class=o>=</span> <span class=mi>0</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=k>static</span> <span class=kt>float</span> <span class=n>derivative</span> <span class=o>=</span> <span class=mi>0</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=k>static</span> <span class=kt>float</span> <span class=n>last_error</span> <span class=o>=</span> <span class=mi>0</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=k>static</span> <span class=kt>float</span> <span class=n>error</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=k>static</span> <span class=kt>float</span> <span class=n>delta</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=k>static</span> <span class=kt>float</span> <span class=n>last_time</span> <span class=o>=</span> <span class=o>*</span><span class=n>cur_loop_timestamp</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=k>static</span> <span class=k>const</span> <span class=kt>float</span> <span class=n>min_water_temp</span> <span class=o>=</span> <span class=mf>5.0</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=k>static</span> <span class=k>const</span> <span class=kt>float</span> <span class=n>max_water_temp</span> <span class=o>=</span> <span class=mf>40.0</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=k>static</span> <span class=k>const</span> <span class=kt>uint8_t</span> <span class=n>min_fan_speed</span> <span class=o>=</span> <span class=mi>90</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=k>static</span> <span class=k>const</span> <span class=kt>uint8_t</span> <span class=n>max_fan_speed</span> <span class=o>=</span> <span class=mi>255</span><span class=p>;</span>
</span></span><span class=line><span class=cl>
</span></span><span class=line><span class=cl> <span class=k>static</span> <span class=k>const</span> <span class=kt>uint8_t</span> <span class=n>temp_target_offset</span> <span class=o>=</span> <span class=mi>4</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=k>static</span> <span class=k>const</span> <span class=kt>uint8_t</span> <span class=n>fan_offset</span> <span class=o>=</span> <span class=mi>10</span><span class=p>;</span>
</span></span><span class=line><span class=cl>
</span></span><span class=line><span class=cl> <span class=k>static</span> <span class=kt>int16_t</span> <span class=n>fan_speed</span><span class=p>;</span>
</span></span><span class=line><span class=cl>
</span></span><span class=line><span class=cl> <span class=k>if</span> <span class=p>(</span><span class=o>*</span><span class=n>water_temp</span> <span class=o>&gt;=</span> <span class=n>max_water_temp</span><span class=p>)</span> <span class=p>{</span>
</span></span><span class=line><span class=cl> <span class=k>return</span> <span class=n>max_fan_speed</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=p>}</span>
</span></span><span class=line><span class=cl>
</span></span><span class=line><span class=cl> <span class=k>else</span> <span class=k>if</span> <span class=p>(</span><span class=o>*</span><span class=n>water_temp</span> <span class=o>&lt;=</span> <span class=n>min_water_temp</span><span class=p>)</span> <span class=p>{</span>
</span></span><span class=line><span class=cl> <span class=k>return</span> <span class=mi>0</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=p>}</span>
</span></span><span class=line><span class=cl>
</span></span><span class=line><span class=cl> <span class=k>else</span> <span class=p>{</span>
</span></span><span class=line><span class=cl> <span class=n>error</span> <span class=o>=</span> <span class=o>*</span><span class=n>water_temp</span> <span class=o>-</span> <span class=n>min</span><span class=p>(</span><span class=o>*</span><span class=n>air_temp</span> <span class=o>+</span> <span class=n>temp_target_offset</span><span class=p>,</span> <span class=n>max_water_temp</span><span class=p>);</span>
</span></span><span class=line><span class=cl> <span class=n>delta</span> <span class=o>=</span> <span class=o>*</span><span class=n>cur_loop_timestamp</span> <span class=o>-</span> <span class=n>last_time</span><span class=p>;</span>
</span></span><span class=line><span class=cl>
</span></span><span class=line><span class=cl> <span class=c1>//mitigate unlimited integral windup
</span></span></span><span class=line><span class=cl><span class=c1></span> <span class=k>if</span> <span class=p>(</span><span class=n>fan_speed</span> <span class=o>==</span> <span class=n>max_fan_speed</span><span class=p>)</span> <span class=p>{</span>
</span></span><span class=line><span class=cl> <span class=n>integral</span> <span class=o>+=</span> <span class=n>error</span> <span class=o>*</span> <span class=n>delta</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=p>}</span>
</span></span><span class=line><span class=cl>
</span></span><span class=line><span class=cl> <span class=k>if</span> <span class=p>(</span><span class=o>*</span><span class=n>air_temp</span> <span class=o>+</span> <span class=n>temp_target_offset</span> <span class=o>&gt;</span> <span class=o>*</span><span class=n>water_temp</span> <span class=o>-</span> <span class=mi>1</span><span class=p>)</span> <span class=p>{</span>
</span></span><span class=line><span class=cl> <span class=n>integral</span> <span class=o>=</span> <span class=mi>0</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=p>}</span>
</span></span><span class=line><span class=cl>
</span></span><span class=line><span class=cl> <span class=n>derivative</span> <span class=o>=</span> <span class=p>(</span><span class=n>error</span> <span class=o>-</span> <span class=n>last_error</span><span class=p>)</span> <span class=o>/</span> <span class=n>delta</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=n>fan_speed</span> <span class=o>=</span> <span class=n>round</span><span class=p>(</span><span class=n>constrain</span><span class=p>(</span><span class=n>min_fan_speed</span> <span class=o>+</span> <span class=n>fan_offset</span> <span class=o>+</span> <span class=p>(</span><span class=n>kp</span> <span class=o>*</span> <span class=n>error</span> <span class=o>+</span> <span class=n>ki</span> <span class=o>*</span> <span class=n>integral</span> <span class=o>+</span> <span class=n>kd</span> <span class=o>*</span> <span class=n>derivative</span><span class=p>),</span> <span class=n>min_fan_speed</span><span class=p>,</span> <span class=n>max_fan_speed</span><span class=p>));</span>
</span></span><span class=line><span class=cl> <span class=n>last_error</span> <span class=o>=</span> <span class=n>error</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=k>return</span> <span class=n>fan_speed</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=p>}</span>
</span></span><span class=line><span class=cl><span class=p>}</span>
</span></span></code></pre></div><p>Occasionally, the temperature probes as well as the HX711 return spurious readings that cause poor behavior, such as crashing the Arduino. In particular, the temperature probes will sometimes return -127, which caused my PID algorithm to crash the Arduino for reasons I could not divine.</p><p>For the temperature probes, I simply ignore the one problematic result that I&rsquo;ve observed.</p><div class=highlight><pre tabindex=0 class=chroma><code class=language-c++ data-lang=c++><span class=line><span class=cl><span class=n>new_water_temp</span> <span class=o>=</span> <span class=n>sensors</span><span class=p>.</span><span class=n>getTempC</span><span class=p>(</span><span class=n>water_therm</span><span class=p>);</span>
</span></span><span class=line><span class=cl><span class=n>new_air_temp</span> <span class=o>=</span> <span class=n>sensors</span><span class=p>.</span><span class=n>getTempC</span><span class=p>(</span><span class=n>air_therm</span><span class=p>);</span>
</span></span><span class=line><span class=cl>
</span></span><span class=line><span class=cl><span class=k>if</span> <span class=p>(</span><span class=n>new_water_temp</span> <span class=o>!=</span> <span class=o>-</span><span class=mi>127</span><span class=p>)</span> <span class=p>{</span>
</span></span><span class=line><span class=cl> <span class=n>water_temp</span> <span class=o>=</span> <span class=n>new_water_temp</span><span class=p>;</span>
</span></span><span class=line><span class=cl><span class=p>}</span>
</span></span><span class=line><span class=cl>
</span></span><span class=line><span class=cl><span class=k>if</span> <span class=p>(</span><span class=n>new_air_temp</span> <span class=o>!=</span> <span class=o>-</span><span class=mi>127</span><span class=p>)</span> <span class=p>{</span>
</span></span><span class=line><span class=cl> <span class=n>air_temp</span> <span class=o>=</span> <span class=n>new_air_temp</span><span class=p>;</span>
</span></span><span class=line><span class=cl><span class=p>}</span>
</span></span><span class=line><span class=cl>
</span></span><span class=line><span class=cl><span class=n>sensors</span><span class=p>.</span><span class=n>requestTemperatures</span><span class=p>();</span>
</span></span></code></pre></div><p>In case of any other freezing/crashing issues, I also enabled the watchdog timer for 2 seconds. So, if, for some reason, it does freeze/crash, it should self reset after 2 seconds. It seems to be working, although I guess time will tell in the long-term. I haven&rsquo;t experienced any operation issues since I added it over a month ago. The other concern is undefined behavior when the timer overflows. The Uno only has a 32-bit timer, so it will overflow around 50 days of uptime. This function pre-emptively resets it.</p><div class=highlight><pre tabindex=0 class=chroma><code class=language-c++ data-lang=c++><span class=line><span class=cl><span class=c1>//we will use this function to periodically self-reset to avoid timer overflows
</span></span></span><span class=line><span class=cl><span class=c1></span><span class=kt>void</span><span class=p>(</span><span class=o>*</span> <span class=n>resetFunc</span><span class=p>)</span> <span class=p>(</span><span class=kt>void</span><span class=p>)</span> <span class=o>=</span> <span class=mi>0</span><span class=p>;</span>
</span></span><span class=line><span class=cl>
</span></span><span class=line><span class=cl><span class=p>...</span>
</span></span><span class=line><span class=cl>
</span></span><span class=line><span class=cl><span class=c1>//reset the system when approaching timer overflow
</span></span></span><span class=line><span class=cl><span class=c1></span><span class=k>if</span> <span class=p>(</span><span class=n>cur_loop_timestamp</span> <span class=o>&gt;=</span> <span class=mi>4000000000</span><span class=p>)</span> <span class=p>{</span>
</span></span><span class=line><span class=cl> <span class=n>resetFunc</span><span class=p>();</span>
</span></span><span class=line><span class=cl><span class=p>}</span>
</span></span></code></pre></div><p>In addition to the temperature probe problem, the HX711 occasionally returns wildly wrong results that need to be filtered out. To compensate for that, if the pressure dips below the threshold, I wait one second, and if the pressure is still below the threshold, I then begin pumping. This check happens approximately ten times per second, as the default behavior of the HX711 board that I have is to run in 10hz mode. I&rsquo;m not sure if the issue springs from some kind of interference with the tachometer interrupts messing up the signaling timing, or if I&rsquo;m misunderstanding the correct way to sample the HX711 over time.</p><div class=highlight><pre tabindex=0 class=chroma><code class=language-c++ data-lang=c++><span class=line><span class=cl><span class=k>if</span> <span class=p>(</span><span class=n>cur_loop_timestamp</span> <span class=o>-</span> <span class=n>last_pressure_check</span> <span class=o>&gt;=</span> <span class=mi>100</span><span class=p>)</span> <span class=p>{</span>
</span></span><span class=line><span class=cl> <span class=n>loop_pressure</span> <span class=o>=</span> <span class=n>pressure_raw_to_mbar</span><span class=p>(</span><span class=n>hx711</span><span class=p>.</span><span class=n>readChannelRaw</span><span class=p>(</span><span class=n>CHAN_A_GAIN_64</span><span class=p>));</span>
</span></span><span class=line><span class=cl>
</span></span><span class=line><span class=cl> <span class=k>if</span> <span class=p>(</span><span class=n>sucking</span> <span class=o>==</span> <span class=nb>false</span><span class=p>)</span> <span class=p>{</span>
</span></span><span class=line><span class=cl> <span class=k>if</span> <span class=p>(</span><span class=n>loop_pressure</span> <span class=o>&gt;</span> <span class=n>low_pressure_threshold</span><span class=p>)</span> <span class=p>{</span>
</span></span><span class=line><span class=cl> <span class=k>if</span> <span class=p>(</span><span class=n>checking_low_pressure</span> <span class=o>==</span> <span class=nb>false</span><span class=p>)</span> <span class=p>{</span>
</span></span><span class=line><span class=cl> <span class=n>checking_low_pressure</span> <span class=o>=</span> <span class=nb>true</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=n>low_pressure_confirmation_timestamp</span> <span class=o>=</span> <span class=n>cur_loop_timestamp</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=p>}</span>
</span></span><span class=line><span class=cl> <span class=k>if</span> <span class=p>(</span><span class=n>cur_loop_timestamp</span> <span class=o>-</span> <span class=n>low_pressure_confirmation_timestamp</span> <span class=o>&gt;=</span> <span class=mi>1000</span><span class=p>)</span> <span class=p>{</span>
</span></span><span class=line><span class=cl> <span class=n>digitalWrite</span><span class=p>(</span><span class=n>pump_relay</span><span class=p>,</span> <span class=n>HIGH</span><span class=p>);</span>
</span></span><span class=line><span class=cl> <span class=n>digitalWrite</span><span class=p>(</span><span class=n>solenoid_relay</span><span class=p>,</span> <span class=n>HIGH</span><span class=p>);</span>
</span></span><span class=line><span class=cl> <span class=n>sucking</span> <span class=o>=</span> <span class=nb>true</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=n>checking_low_pressure</span> <span class=o>=</span> <span class=nb>false</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=p>}</span>
</span></span><span class=line><span class=cl> <span class=p>}</span>
</span></span><span class=line><span class=cl>
</span></span><span class=line><span class=cl> <span class=k>else</span> <span class=nf>if</span> <span class=p>(</span><span class=n>cur_loop_timestamp</span> <span class=o>-</span> <span class=n>low_pressure_confirmation_timestamp</span> <span class=o>&gt;=</span> <span class=mi>1000</span><span class=p>)</span> <span class=p>{</span>
</span></span><span class=line><span class=cl> <span class=n>checking_low_pressure</span> <span class=o>=</span> <span class=nb>false</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=p>}</span>
</span></span><span class=line><span class=cl> <span class=p>}</span>
</span></span><span class=line><span class=cl>
</span></span><span class=line><span class=cl> <span class=k>else</span> <span class=p>{</span>
</span></span><span class=line><span class=cl> <span class=k>if</span> <span class=p>(</span><span class=n>loop_pressure</span> <span class=o>&lt;</span> <span class=n>high_pressure_threshold</span><span class=p>)</span> <span class=p>{</span>
</span></span><span class=line><span class=cl> <span class=n>digitalWrite</span><span class=p>(</span><span class=n>pump_relay</span><span class=p>,</span> <span class=n>LOW</span><span class=p>);</span>
</span></span><span class=line><span class=cl> <span class=n>digitalWrite</span><span class=p>(</span><span class=n>solenoid_relay</span><span class=p>,</span> <span class=n>LOW</span><span class=p>);</span>
</span></span><span class=line><span class=cl> <span class=n>sucking</span> <span class=o>=</span> <span class=nb>false</span><span class=p>;</span>
</span></span><span class=line><span class=cl> <span class=p>}</span>
</span></span><span class=line><span class=cl> <span class=p>}</span>
</span></span><span class=line><span class=cl> <span class=n>last_pressure_check</span> <span class=o>=</span> <span class=n>cur_loop_timestamp</span><span class=p>;</span>
</span></span><span class=line><span class=cl><span class=p>}</span>
</span></span></code></pre></div><p>Currently, my server-side software is incomplete. It&rsquo;s just a brute-force JSON-over-serial reader written in Python, that I glance at from time to time. I plan write a Zabbix bridge, and have that manage the monitoring, alerts, and reactions to catostraphic events, once I have Zabbix properly setup for my systems&mldr; But that hasn&rsquo;t happened just yet. I don&rsquo;t expect it to be a particularly interesting event, but if anything comes up I might write a post about it.</p><h2 class="relative group">Other Thoughts?<div id=other-thoughts class=anchor></div><span class="absolute top-0 w-6 transition-opacity opacity-0 ltr:-left-6 rtl:-right-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style=text-decoration-line:none!important href=#other-thoughts aria-label=Anchor>#</a></span></h2><p>The Arduino&rsquo;s software hadn&rsquo;t been 100% finalized when I took the below pictures. The control box does have a lid now, and all the cable management is a lot cleaner&mldr; Promise!</p><hr><div id=gallery-e7320abbafcc639029162f7da034f020 class=gallery><div class="panzoom-container grid-w50"><img class="my-0 rounded-md zoomable" src=/posts/watercooling-homelab/images/compressed/complete_a.webp srcset="/posts/watercooling-homelab/images/compressed/complete_a_hu2371455810172732269.webp 330w,
/posts/watercooling-homelab/images/compressed/complete_a_hu11784685921316552556.webp 660w,
/posts/watercooling-homelab/images/compressed/complete_a_hu2347091142476872481.webp 1320w,
/posts/watercooling-homelab/images/compressed/complete_a.webp 2x" alt="Front-ish view of the complete rack assembly." data-src=images/compressed/complete_a.webp></div><div class="panzoom-container grid-w50"><img class="my-0 rounded-md zoomable" src=/posts/watercooling-homelab/images/compressed/complete_b.webp srcset="/posts/watercooling-homelab/images/compressed/complete_b_hu10444887637626475779.webp 330w,
/posts/watercooling-homelab/images/compressed/complete_b_hu12966552216401371125.webp 660w,
/posts/watercooling-homelab/images/compressed/complete_b_hu10235357959924345043.webp 1320w,
/posts/watercooling-homelab/images/compressed/complete_b.webp 2x" alt="Back-ish view of the complete rack assembly." data-src=images/compressed/complete_b.webp></div></div><figcaption class=caption>Everything installed and working!</figcaption><hr><p>When I was testing it, I had an incident where the Arduino crashed, which means the fans stopped&mldr; That&rsquo;s a big drawback with the motor controller that I have, it fails off instead of on. But, I haven&rsquo;t experienced any more issues after adding those software fixes. At that time, I was running a full GPU workload&mldr; The water temperature exceeded 70*C. It happened at night, and I have no idea how long it ran like that&mldr; Hours. Pretty scary stuff, but it all came out alright.</p><p>This project had a lot of firsts for me. It was the first time I&rsquo;ve done any kind of embedded-adjacent development beyond &ldquo;ooooo look at the blinky light, oooooooo it turns off when you press the button, wwaaaow&rdquo;, and the first time I&rsquo;d designed something with so many individual parts. I&rsquo;ve never worked with air pumps, solenoids, or pressure sensing before, nor had to debug issues like the lack of flyback diodes.</p><p>The biggest mistake I made was using that stupid battery box. It&rsquo;s steel, and I don&rsquo;t have the tools or experience to work with steel in the way that I intended to. I thought it would look cool, and it does, but if I did it again, I&rsquo;d use a generic aluminum or plastic project box instead, because it took two entire days plus waiting for new drill bits that can actually cut through it.</p><p>If I were to ever take it apart again, I&rsquo;d add a passthrough for the SPI header, and/or an external reset button. I should have gotten a physical display of some type that could show the sensors and debug info on the device itself without being connected to another device to readout the data.</p><p>I&rsquo;d like to get a second pump, for redundancy&rsquo;s sake and to increase the flow rate. But it&rsquo;s going to be such a pain to install that I feel like I&rsquo;m never going to bother to do it, unless the current pump fails, or I add more components to be cooled and the flow is adversely affected. I was slightly concerned about the evaporation rate of the liquid via the vacuum tank, and that I&rsquo;d need to add some kind of fluid level detection system, but there&rsquo;s been no noticeable loss thus far. Now that I know the pump turns on so infrequently, I can&rsquo;t imagine that it&rsquo;s going to need to be topped up anytime soon.</p><p>In terms of value&mldr; This was unbelievably bad. Buying tower coolers would have allowed the CPUs to run without throttling, and buying another GPU would have overpowered any benefits gained by NVLINK. I haven&rsquo;t tallied up exactly how much I spent on it, but it was at least $1000, including buying new tools and excess materials that I haven&rsquo;t fully used, and excluding the original cost of re-using some parts I already had. I&rsquo;ve added risk, maintenance overhead, and pain whenever I swap out hardware in the future. Custom watercooling<sup id=fnref:4><a href=#fn:4 class=footnote-ref role=doc-noteref>4</a></sup> is an ongoing abusive relationship between your fingertips and your ego, or your fascination for slightly more optimized numbers on a screen&mldr; But I&rsquo;d do it again in a heartbeat, because it was fun.</p><div class=footnotes role=doc-endnotes><hr><ol><li id=fn:1><p>In modern, post-Turing cards, that is. Please stop buying mesozoic-era Kepler/Maxwell Quadros and Teslas just because they have VRAM. There&rsquo;s a reason they&rsquo;re going for like, $20, and if you paid more for anything from that era, I&rsquo;m sorry. Electrical costs are a thing, and your life is worth more than waiting for any meaningful, current-year work to happen on those decrepit e-waste cards. I feel even worse for you if you got tricked into buying one of those &ldquo;24GB&rdquo; or &ldquo;16GB&rdquo; cards that are actually 2x12GB and 2x8GB. You can make an argument for Volta, but only if you&rsquo;re doing some deranged pure FP64 stuff. Consumer Turing and newer are faster at everything else! And if you&rsquo;re buying them for HW-accel encode&mldr; The quality is awful compared to any Intel ARC card. Buy one of those instead.&#160;<a href=#fnref:1 class=footnote-backref role=doc-backlink>&#8617;&#xfe0e;</a></p></li><li id=fn:2><p>I don&rsquo;t understand why people don&rsquo;t trust the manufacturer specifications when it comes to silicon temperature limits, beyond unfounded conspiracy nonsense around planned destruction/obselence. In terms of Intel server SKUs, you find that the throttling temp is <em>higher</em> than on consumer SKUs, despite the higher reliability demanded by the enterprise market&mldr; I&rsquo;m assuming that this is due to reduced hotspot variance thanks to generally lower voltage spread from lower boosting clock speeds. On enterprise SKUs which are focused on single threaded performance, the throttling temp is typically lower than those without the ability to boost as high. If you have evidence to the contrary, let me know.&#160;<a href=#fnref:2 class=footnote-backref role=doc-backlink>&#8617;&#xfe0e;</a></p></li><li id=fn:3><p>NVIDIA overclocking on Linux is awful. There&rsquo;s no way to edit the voltage-frequency curve through the Linux drivers. You can only set the offset. So you can&rsquo;t really make use of dynamic boosting if you want to undervolt. I have the max clock speed clamped at 1800MHz with a core offset to emulate undervolting as you would do on Windows, but it&rsquo;s hard to say if I&rsquo;m getting the peak performance that I could be getting at whatever the core voltage is under these circumstances - because NVIDIA&rsquo;s Linux drivers ALSO don&rsquo;t report that. VRAM temperature? Nope. VRM? Nope. Hotspot? Nope. You better hope your card works with NVML, too, because otherwise you&rsquo;re going to have to mess around with X to use nvidia-settings.&#160;<a href=#fnref:3 class=footnote-backref role=doc-backlink>&#8617;&#xfe0e;</a></p></li><li id=fn:4><p>Stop calling them open loops. Open loop systems exchange fluid with the environment. Unless you&rsquo;re getting your water out of the sink and flushing it right down the drain, your system is not open loop. It&rsquo;s closed. I don&rsquo;t understand why people call custom loops &ldquo;open&rdquo; and AIOs &ldquo;closed&rdquo; when their modes of operation are identical. It&rsquo;s just plain wrong.&#160;<a href=#fnref:4 class=footnote-backref role=doc-backlink>&#8617;&#xfe0e;</a></p></li></ol></div><strong class="block mt-8"><a target=_blank class="m-1 rounded bg-neutral-300 p-1.5 text-neutral-700 hover:bg-primary-500 hover:text-neutral dark:bg-neutral-700 dark:text-neutral-300 dark:hover:bg-primary-400 dark:hover:text-neutral-800" href="mailto:rawhide@neet.works?subject=Reply%20to%20Watercooling%20My%20Homelab">Reply by Email</a></strong></div></div><script>var oid="views_posts/watercooling-my-homelab/index.md",oid_likes="likes_posts/watercooling-my-homelab/index.md"</script><script type=text/javascript src=/js/page.min.0860cf4e04fa2d72cc33ddba263083464d48f67de06114529043cb4623319efed4f484fd7f1730df5abea0e2da6f3538855634081d02f2d6e920b956f063e823.js integrity="sha512-CGDPTgT6LXLMM926JjCDRk1I9n3gYRRSkEPLRiMxnv7U9IT9fxcw31q+oOLabzU4hVY0CB0C8tbpILlW8GPoIw=="></script></section><footer class="pt-8 max-w-prose print:hidden"><div class=pt-8><hr class="border-dotted border-neutral-300 dark:border-neutral-600"><div class="flex justify-between pt-3"><span></span><span><a class="flex text-right group ml-3" href=/posts/maxcloudon-pcie-bifurcation-riser-review/><span class="flex flex-col"><span class="mt-[0.1rem] leading-6 group-hover:underline group-hover:decoration-primary-500">MaxCloudON PCIe Bifurcation Riser Review</span>
<span class="mt-[0.1rem] text-xs text-neutral-500 dark:text-neutral-400"><time datetime=2025-01-19T07:32:31+00:00>January 19 2025</time>
</span></span><span class="ml-3 text-neutral-700 group-hover:text-primary-600 ltr:inline rtl:hidden dark:text-neutral dark:group-hover:text-primary-400">&rarr;</span>
<span class="mr-3 text-neutral-700 group-hover:text-primary-600 ltr:hidden rtl:inline dark:text-neutral dark:group-hover:text-primary-400">&larr;</span></a></span></div></div></footer></article><div id=top-scroller class="pointer-events-none absolute top-[110vh] bottom-0 w-12 ltr:right-0 rtl:left-0"><a href=#the-top class="pointer-events-auto sticky top-[calc(100vh-5.5rem)] flex h-12 w-12 mb-16 items-center justify-center rounded-full bg-neutral/50 text-xl text-neutral-700 hover:text-primary-600 dark:bg-neutral-800/50 dark:text-neutral dark:hover:text-primary-400" aria-label="Scroll to top" title="Scroll to top">&uarr;</a></div></main><footer id=site-footer class="py-10 print:hidden"><div class="flex items-center justify-between"><p class="text-sm text-neutral-500 dark:text-neutral-400"><em>© 2025 rawhide kobayashi | <strong>Original content</strong> licensed under <strong><a href=https://creativecommons.org/licenses/by-sa/4.0/ target=_blank>CC BY-SA 4.0</a></strong> unless otherwise noted.</em></p><p class="text-xs text-neutral-500 dark:text-neutral-400">Powered by <a class="hover:underline hover:decoration-primary-400 hover:text-primary-500" href=https://gohugo.io/ target=_blank rel="noopener noreferrer">Hugo</a> & <a class="hover:underline hover:decoration-primary-400 hover:text-primary-500" href=https://blowfish.page/ target=_blank rel="noopener noreferrer">Blowfish</a></p></div><script></script><script type=text/javascript src=/js/process.min.ee03488f19c93c2efb199e2e3014ea5f3cb2ce7d45154adb3399a158cac27ca52831db249ede5bb602700ef87eb02434139de0858af1818ab0fb4182472204a4.js integrity="sha512-7gNIjxnJPC77GZ4uMBTqXzyyzn1FFUrbM5mhWMrCfKUoMdsknt5btgJwDvh+sCQ0E53ghYrxgYqw+0GCRyIEpA=="></script></footer><div id=search-wrapper class="invisible fixed inset-0 flex h-screen w-screen cursor-default flex-col bg-neutral-500/50 p-4 backdrop-blur-sm dark:bg-neutral-900/50 sm:p-6 md:p-[10vh] lg:p-[12vh]" data-url=https://blog.neet.works/ style=z-index:500><div id=search-modal class="flex flex-col w-full max-w-3xl min-h-0 mx-auto border rounded-md shadow-lg top-20 border-neutral-200 bg-neutral dark:border-neutral-700 dark:bg-neutral-800"><header class="relative z-10 flex items-center justify-between flex-none px-2"><form class="flex items-center flex-auto min-w-0"><div class="flex items-center justify-center w-8 h-8 text-neutral-400"><span class="relative block icon"><svg aria-hidden="true" focusable="false" data-prefix="fas" data-icon="search" class="svg-inline--fa fa-search fa-w-16" role="img" viewBox="0 0 512 512"><path fill="currentcolor" d="M505 442.7 405.3 343c-4.5-4.5-10.6-7-17-7H372c27.6-35.3 44-79.7 44-128C416 93.1 322.9.0 208 0S0 93.1.0 208s93.1 208 208 208c48.3.0 92.7-16.4 128-44v16.3c0 6.4 2.5 12.5 7 17l99.7 99.7c9.4 9.4 24.6 9.4 33.9.0l28.3-28.3c9.4-9.4 9.4-24.6.1-34zM208 336c-70.7.0-128-57.2-128-128 0-70.7 57.2-128 128-128 70.7.0 128 57.2 128 128 0 70.7-57.2 128-128 128z"/></svg></span></div><input type=search id=search-query class="flex flex-auto h-12 mx-1 bg-transparent appearance-none focus:outline-dotted focus:outline-2 focus:outline-transparent" placeholder=Search tabindex=0></form><button id=close-search-button class="flex items-center justify-center w-8 h-8 text-neutral-700 hover:text-primary-600 dark:text-neutral dark:hover:text-primary-400" title="Close (Esc)">
<span class="relative block icon"><svg viewBox="0 0 320 512"><path fill="currentcolor" d="M310.6 361.4c12.5 12.5 12.5 32.75.0 45.25C304.4 412.9 296.2 416 288 416s-16.38-3.125-22.62-9.375L160 301.3 54.63 406.6C48.38 412.9 40.19 416 32 416S15.63 412.9 9.375 406.6c-12.5-12.5-12.5-32.75.0-45.25l105.4-105.4L9.375 150.6c-12.5-12.5-12.5-32.75.0-45.25s32.75-12.5 45.25.0L160 210.8l105.4-105.4c12.5-12.5 32.75-12.5 45.25.0s12.5 32.75.0 45.25l-105.4 105.4L310.6 361.4z"/></svg></span></button></header><section class="flex-auto px-2 overflow-auto"><ul id=search-results></ul></section></div></div></div></body></html>
Reference in New Issue View Git Blame Copy Permalink