https://blog.wu-boy.com/tags/grafana/Recent content in Grafana on 小惡魔 - AppleBOYHugoen-usappleboy.tw@gmail.com (Appleboy)appleboy.tw@gmail.com (Appleboy)Sun, 16 May 2021 00:57:45 +0000https://blog.wu-boy.com/2021/05/mongodb-performance-tunning/Sun, 16 May 2021 00:57:45 +0000appleboy.tw@gmail.com (Appleboy)https://blog.wu-boy.com/2021/05/mongodb-performance-tunning/<p><a href="https://lh3.googleusercontent.com/DZKO3gMs5RhQ0-uGU2Y-uaTsb7HKCJU3lH91uggni5HA-fpDMqgvKPwHRwuo-jlCbAJZYyY9TKKovtCDT7OFgiclb2VYz58HwmDeHUX6FjlwfnuTkaTZxYudTIiuJ6yWsuNu2vs1vTQ=w1920-h1080" title="mongodb"><img src="https://lh3.googleusercontent.com/DZKO3gMs5RhQ0-uGU2Y-uaTsb7HKCJU3lH91uggni5HA-fpDMqgvKPwHRwuo-jlCbAJZYyY9TKKovtCDT7OFgiclb2VYz58HwmDeHUX6FjlwfnuTkaTZxYudTIiuJ6yWsuNu2vs1vTQ=w1920-h1080" alt="mongodb" title="mongodb"></a></p> <p>最近剛好在實作 <a href="https://prometheus.io/">Prometheus</a> + <a href="https://grafana.com/">Grafana</a> 的時候，對 MongoDB 做了容器 CPU 使用率 (<code>container_cpu_usage_seconds_total</code>) 的監控，Metrics 寫法如下:</p> <div class="highlight"><div style="background-color:#eed;-moz-tab-size:2;-o-tab-size:2;tab-size:2;-webkit-text-size-adjust:none;"> <table style="border-spacing:0;padding:0;margin:0;border:0;"><tr><td style="vertical-align:top;padding:0;margin:0;border:0;"> <pre tabindex="0" style="background-color:#eed;-moz-tab-size:2;-o-tab-size:2;tab-size:2;-webkit-text-size-adjust:none;"><code><span style="white-space:pre;-webkit-user-select:none;user-select:none;margin-right:0.4em;padding:0 0.4em 0 0.4em;color:#7f7f7f" id="hl-0-1"><a style="outline:none;text-decoration:none;color:inherit" href="#hl-0-1">1</a> </span><span style="white-space:pre;-webkit-user-select:none;user-select:none;margin-right:0.4em;padding:0 0.4em 0 0.4em;color:#7f7f7f" id="hl-0-2"><a style="outline:none;text-decoration:none;color:inherit" href="#hl-0-2">2</a> </span><span style="white-space:pre;-webkit-user-select:none;user-select:none;margin-right:0.4em;padding:0 0.4em 0 0.4em;color:#7f7f7f" id="hl-0-3"><a style="outline:none;text-decoration:none;color:inherit" href="#hl-0-3">3</a> </span></code></pre></td> <td style="vertical-align:top;padding:0;margin:0;border:0;;width:100%"> <pre tabindex="0" style="background-color:#eed;-moz-tab-size:2;-o-tab-size:2;tab-size:2;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>sum( </span></span><span style="display:flex;"><span> rate(container_cpu_usage_seconds_total{name!~<span style="color:#cd5555">&#34;(^</span>$<span style="color:#cd5555">|^0_.*)&#34;</span>}[1m])) </span></span><span style="display:flex;"><span>by (name) </span></span></code></pre></td></tr></table> </div> </div><p>從上面的 Metrics 可以拉長時間來看，會發現專案的 MongoDB 非常不穩定，起起伏伏，這時候就需要來看看資料庫到底哪邊慢，以及看看哪個語法造成 CPU 飆高？</p> <p><img src="https://lh3.googleusercontent.com/FbcbJ75SVSEmNb94X6z9JsjkhKmuzjEGUesnTVxwcP2SGYWJpblQaD5ks02brR9kP9HYqP7KpbQAaoa7RUuBWi8EnXdN2eTCzekyGVmKAY4ltnmEnNrWerAzZkHIp9gGKieO71WUhJk=w1920-h1080" alt=""></p> <p>接著為了看 MongoDB 的 Log 紀錄，把 Grafana 推出的 <a href="https://grafana.com/oss/loki/">Loki</a>，也導入專案系統，將容器所有的 Log 都導向 Loki，底下可以看看 docker-compose 將 Log 輸出到 loki</p> <div class="highlight"><div style="background-color:#eed;-moz-tab-size:2;-o-tab-size:2;tab-size:2;-webkit-text-size-adjust:none;"> <table style="border-spacing:0;padding:0;margin:0;border:0;"><tr><td style="vertical-align:top;padding:0;margin:0;border:0;"> <pre tabindex="0" style="background-color:#eed;-moz-tab-size:2;-o-tab-size:2;tab-size:2;-webkit-text-size-adjust:none;"><code><span style="white-space:pre;-webkit-user-select:none;user-select:none;margin-right:0.4em;padding:0 0.4em 0 0.4em;color:#7f7f7f" id="hl-1-1"><a style="outline:none;text-decoration:none;color:inherit" href="#hl-1-1">1</a> </span><span style="white-space:pre;-webkit-user-select:none;user-select:none;margin-right:0.4em;padding:0 0.4em 0 0.4em;color:#7f7f7f" id="hl-1-2"><a style="outline:none;text-decoration:none;color:inherit" href="#hl-1-2">2</a> </span><span style="white-space:pre;-webkit-user-select:none;user-select:none;margin-right:0.4em;padding:0 0.4em 0 0.4em;color:#7f7f7f" id="hl-1-3"><a style="outline:none;text-decoration:none;color:inherit" href="#hl-1-3">3</a> </span><span style="white-space:pre;-webkit-user-select:none;user-select:none;margin-right:0.4em;padding:0 0.4em 0 0.4em;color:#7f7f7f" id="hl-1-4"><a style="outline:none;text-decoration:none;color:inherit" href="#hl-1-4">4</a> </span><span style="white-space:pre;-webkit-user-select:none;user-select:none;margin-right:0.4em;padding:0 0.4em 0 0.4em;color:#7f7f7f" id="hl-1-5"><a style="outline:none;text-decoration:none;color:inherit" href="#hl-1-5">5</a> </span><span style="white-space:pre;-webkit-user-select:none;user-select:none;margin-right:0.4em;padding:0 0.4em 0 0.4em;color:#7f7f7f" id="hl-1-6"><a style="outline:none;text-decoration:none;color:inherit" href="#hl-1-6">6</a> </span><span style="white-space:pre;-webkit-user-select:none;user-select:none;margin-right:0.4em;padding:0 0.4em 0 0.4em;color:#7f7f7f" id="hl-1-7"><a style="outline:none;text-decoration:none;color:inherit" href="#hl-1-7">7</a> </span></code></pre></td> <td style="vertical-align:top;padding:0;margin:0;border:0;;width:100%"> <pre tabindex="0" style="background-color:#eed;-moz-tab-size:2;-o-tab-size:2;tab-size:2;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span> logging: </span></span><span style="display:flex;"><span> driver: loki </span></span><span style="display:flex;"><span> options: </span></span><span style="display:flex;"><span> loki-url: &#34;http://xxxxxxx/loki/api/v1/push&#34; </span></span><span style="display:flex;"><span> loki-retries: &#34;5&#34; </span></span><span style="display:flex;"><span> loki-batch-size: &#34;400&#34; </span></span><span style="display:flex;"><span> loki-external-labels: &#34;environment=production,project=mongo&#34; </span></span></code></pre></td></tr></table> </div> </div><p><img src="https://lh3.googleusercontent.com/TooE3Q49jzI_FpbjXm4b3A_9aho8J4Qws64XmhVzDVbe6NPMCmgYmuw5bMRwnMgmk_lXxNHDU1n6RXwFGvoZvPxLuM6clRJ_ZGRC9S47rvFbm3k9v6v8qaHhC6vqsFkXENQYlRAqKn0=w1920-h1080" alt=""></p> <p>先看看結論，做法其實很簡單，找出相對應 Slow Query，把相關的欄位加上 Index，就可以解決了</p> <p><img src="https://lh3.googleusercontent.com/WgyOX8OOff6KGKlOvSvoNxzDsPGyiXBwPa_PX3O7L9AYSBfQ9VoNAP5s_HkbmMa7rokTnF--ZLnJ4p6oLqoTCV2Gyq7B696SEbTIrGIi1kDVtijVpZlYTklq3qbLFtpKGAQHeXxHE-Q=w1920-h1080" alt=""></p> <p><img src="https://lh3.googleusercontent.com/nzKkL7J5x_LmoqTFYav5kKA8Jkp4E4s8OCOekd-fz2HAeU2ySC3DopumqMIevqelMN_bvw7Ug7BB2f6ZJeubCQzz4w1Uby709NqsqTEkQcJK7IwVkcHt_ZkArRjSlKfZvyWBE6ZBLnY=w1920-h1080" alt=""></p>