[PLUGIN] GPU Statistics


Recommended Posts

9 hours ago, b3rs3rk said:


Look in the original post of this thread and provide the troubleshooting info at the bottom of it.

root@UnRAID:~# nvidia-smi -q -x -i 0
<?xml version="1.0" ?>
<!DOCTYPE nvidia_smi_log SYSTEM "nvsmi_device_v11.dtd">
<nvidia_smi_log>
        <timestamp>Mon Apr  5 10:38:13 2021</timestamp>
        <driver_version>465.19.01</driver_version>
        <cuda_version>11.3</cuda_version>
        <attached_gpus>1</attached_gpus>
        <gpu id="00000000:13:00.0">
                <product_name>NVIDIA Quadro P2200</product_name>
                <product_brand>Quadro</product_brand>
                <display_mode>Disabled</display_mode>
                <display_active>Disabled</display_active>
                <persistence_mode>Disabled</persistence_mode>
                <mig_mode>
                        <current_mig>N/A</current_mig>
                        <pending_mig>N/A</pending_mig>
                </mig_mode>
                <mig_devices>
                        None
                </mig_devices>
                <accounting_mode>Disabled</accounting_mode>
                <accounting_mode_buffer_size>4000</accounting_mode_buffer_size>
                <driver_model>
                        <current_dm>N/A</current_dm>
                        <pending_dm>N/A</pending_dm>
                </driver_model>
                <serial>1324919079153</serial>
                <uuid>GPU-204cd9c8-7a7c-b6cf-f037-d43e7fa9fb10</uuid>
                <minor_number>0</minor_number>
                <vbios_version>86.06.77.00.03</vbios_version>
                <multigpu_board>No</multigpu_board>
                <board_id>0x1300</board_id>
                <gpu_part_number>900-5G420-1700-000</gpu_part_number>
                <inforom_version>
                        <img_version>G420.0500.00.02</img_version>
                        <oem_object>1.1</oem_object>
                        <ecc_object>N/A</ecc_object>
                        <pwr_object>N/A</pwr_object>
                </inforom_version>
                <gpu_operation_mode>
                        <current_gom>N/A</current_gom>
                        <pending_gom>N/A</pending_gom>
                </gpu_operation_mode>
                <gpu_virtualization_mode>
                        <virtualization_mode>Pass-Through</virtualization_mode>
                        <host_vgpu_mode>N/A</host_vgpu_mode>
                </gpu_virtualization_mode>
                <ibmnpu>
                        <relaxed_ordering_mode>N/A</relaxed_ordering_mode>
                </ibmnpu>
                <pci>
                        <pci_bus>13</pci_bus>
                        <pci_device>00</pci_device>
                        <pci_domain>0000</pci_domain>
                        <pci_device_id>1C3110DE</pci_device_id>
                        <pci_bus_id>00000000:13:00.0</pci_bus_id>
                        <pci_sub_system_id>131B10DE</pci_sub_system_id>
                        <pci_gpu_link_info>
                                <pcie_gen>
                                        <max_link_gen>3</max_link_gen>
                                        <current_link_gen>3</current_link_gen>
                                </pcie_gen>
                                <link_widths>
                                        <max_link_width>16x</max_link_width>
                                        <current_link_width>16x</current_link_width>
                                </link_widths>
                        </pci_gpu_link_info>
                        <pci_bridge_chip>
                                <bridge_chip_type>N/A</bridge_chip_type>
                                <bridge_chip_fw>N/A</bridge_chip_fw>
                        </pci_bridge_chip>
                        <replay_counter>0</replay_counter>
                        <replay_rollover_counter>0</replay_rollover_counter>
                        <tx_util>0 KB/s</tx_util>
                        <rx_util>0 KB/s</rx_util>
                </pci>
                <fan_speed>71 %</fan_speed>
                <performance_state>P0</performance_state>
                <clocks_throttle_reasons>
                        <clocks_throttle_reason_gpu_idle>Not Active</clocks_throttle_reason_gpu_idle>
                        <clocks_throttle_reason_applications_clocks_setting>Not Active</clocks_throttle_reason_applications_clocks_setting>
                        <clocks_throttle_reason_sw_power_cap>Not Active</clocks_throttle_reason_sw_power_cap>
                        <clocks_throttle_reason_hw_slowdown>Not Active</clocks_throttle_reason_hw_slowdown>
                        <clocks_throttle_reason_hw_thermal_slowdown>Not Active</clocks_throttle_reason_hw_thermal_slowdown>
                        <clocks_throttle_reason_hw_power_brake_slowdown>Not Active</clocks_throttle_reason_hw_power_brake_slowdown>
                        <clocks_throttle_reason_sync_boost>Not Active</clocks_throttle_reason_sync_boost>
                        <clocks_throttle_reason_sw_thermal_slowdown>Not Active</clocks_throttle_reason_sw_thermal_slowdown>
                        <clocks_throttle_reason_display_clocks_setting>Not Active</clocks_throttle_reason_display_clocks_setting>
                </clocks_throttle_reasons>
                <fb_memory_usage>
                        <total>5059 MiB</total>
                        <used>0 MiB</used>
                        <free>5059 MiB</free>
                </fb_memory_usage>
                <bar1_memory_usage>
                        <total>256 MiB</total>
                        <used>2 MiB</used>
                        <free>254 MiB</free>
                </bar1_memory_usage>
                <compute_mode>Default</compute_mode>
                <utilization>
                        <gpu_util>2 %</gpu_util>
                        <memory_util>0 %</memory_util>
                        <encoder_util>0 %</encoder_util>
                        <decoder_util>0 %</decoder_util>
                </utilization>
                <encoder_stats>
                        <session_count>0</session_count>
                        <average_fps>0</average_fps>
                        <average_latency>0</average_latency>
                </encoder_stats>
                <fbc_stats>
                        <session_count>0</session_count>
                        <average_fps>0</average_fps>
                        <average_latency>0</average_latency>
                </fbc_stats>
                <ecc_mode>
                        <current_ecc>N/A</current_ecc>
                        <pending_ecc>N/A</pending_ecc>
                </ecc_mode>
                <ecc_errors>
                        <volatile>
                                <single_bit>
                                        <device_memory>N/A</device_memory>
                                        <register_file>N/A</register_file>
                                        <l1_cache>N/A</l1_cache>
                                        <l2_cache>N/A</l2_cache>
                                        <texture_memory>N/A</texture_memory>
                                        <texture_shm>N/A</texture_shm>
                                        <cbu>N/A</cbu>
                                        <total>N/A</total>
                                </single_bit>
                                <double_bit>
                                        <device_memory>N/A</device_memory>
                                        <register_file>N/A</register_file>
                                        <l1_cache>N/A</l1_cache>
                                        <l2_cache>N/A</l2_cache>
                                        <texture_memory>N/A</texture_memory>
                                        <texture_shm>N/A</texture_shm>
                                        <cbu>N/A</cbu>
                                        <total>N/A</total>
                                </double_bit>
                        </volatile>
                        <aggregate>
                                <single_bit>
                                        <device_memory>N/A</device_memory>
                                        <register_file>N/A</register_file>
                                        <l1_cache>N/A</l1_cache>
                                        <l2_cache>N/A</l2_cache>
                                        <texture_memory>N/A</texture_memory>
                                        <texture_shm>N/A</texture_shm>
                                        <cbu>N/A</cbu>
                                        <total>N/A</total>
                                </single_bit>
                                <double_bit>
                                        <device_memory>N/A</device_memory>
                                        <register_file>N/A</register_file>
                                        <l1_cache>N/A</l1_cache>
                                        <l2_cache>N/A</l2_cache>
                                        <texture_memory>N/A</texture_memory>
                                        <texture_shm>N/A</texture_shm>
                                        <cbu>N/A</cbu>
                                        <total>N/A</total>
                                </double_bit>
                        </aggregate>
                </ecc_errors>
                <retired_pages>
                        <multiple_single_bit_retirement>
                                <retired_count>N/A</retired_count>
                                <retired_pagelist>N/A</retired_pagelist>
                        </multiple_single_bit_retirement>
                        <double_bit_retirement>
                                <retired_count>N/A</retired_count>
                                <retired_pagelist>N/A</retired_pagelist>
                        </double_bit_retirement>
                        <pending_blacklist>N/A</pending_blacklist>
                        <pending_retirement>N/A</pending_retirement>
                </retired_pages>
                <remapped_rows>N/A</remapped_rows>
                <temperature>
                        <gpu_temp>38 C</gpu_temp>
                        <gpu_temp_max_threshold>102 C</gpu_temp_max_threshold>
                        <gpu_temp_slow_threshold>99 C</gpu_temp_slow_threshold>
                        <gpu_temp_max_gpu_threshold>N/A</gpu_temp_max_gpu_threshold>
                        <gpu_target_temperature>83 C</gpu_target_temperature>
                        <memory_temp>N/A</memory_temp>
                        <gpu_temp_max_mem_threshold>N/A</gpu_temp_max_mem_threshold>
                </temperature>
                <supported_gpu_target_temp>
                        <gpu_target_temp_min>65 C</gpu_target_temp_min>
                        <gpu_target_temp_max>96 C</gpu_target_temp_max>
                </supported_gpu_target_temp>
                <power_readings>
                        <power_state>P0</power_state>
                        <power_management>Supported</power_management>
                        <power_draw>21.72 W</power_draw>
                        <power_limit>75.00 W</power_limit>
                        <default_power_limit>75.00 W</default_power_limit>
                        <enforced_power_limit>75.00 W</enforced_power_limit>
                        <min_power_limit>75.00 W</min_power_limit>
                        <max_power_limit>75.00 W</max_power_limit>
                </power_readings>
                <clocks>
                        <graphics_clock>1050 MHz</graphics_clock>
                        <sm_clock>1050 MHz</sm_clock>
                        <mem_clock>5005 MHz</mem_clock>
                        <video_clock>936 MHz</video_clock>
                </clocks>
                <applications_clocks>
                        <graphics_clock>999 MHz</graphics_clock>
                        <mem_clock>5005 MHz</mem_clock>
                </applications_clocks>
                <default_applications_clocks>
                        <graphics_clock>999 MHz</graphics_clock>
                        <mem_clock>5005 MHz</mem_clock>
                </default_applications_clocks>
                <max_clocks>
                        <graphics_clock>1746 MHz</graphics_clock>
                        <sm_clock>1746 MHz</sm_clock>
                        <mem_clock>5005 MHz</mem_clock>
                        <video_clock>1569 MHz</video_clock>
                </max_clocks>
                <max_customer_boost_clocks>
                        <graphics_clock>1746 MHz</graphics_clock>
                </max_customer_boost_clocks>
                <clock_policy>
                        <auto_boost>N/A</auto_boost>
                        <auto_boost_default>N/A</auto_boost_default>
                </clock_policy>
                <supported_clocks>
                        <supported_mem_clock>
                                <value>5005 MHz</value>
                                <supported_graphics_clock>1746 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1733 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1721 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1708 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1695 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1683 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1670 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1657 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1645 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1632 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1620 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1607 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1594 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1582 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1569 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1556 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1544 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1531 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1518 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1506 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1493 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1480 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1468 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1455 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1442 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1430 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1417 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1404 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1392 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1379 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1366 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1354 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1341 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1328 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1316 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1303 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1290 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1278 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1265 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1252 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1240 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1227 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1215 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1202 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1189 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1177 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1164 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1151 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1139 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1126 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1113 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1101 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1088 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1075 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1063 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1050 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1037 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1025 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1012 MHz</supported_graphics_clock>
                                <supported_graphics_clock>999 MHz</supported_graphics_clock>
                                <supported_graphics_clock>987 MHz</supported_graphics_clock>
                                <supported_graphics_clock>974 MHz</supported_graphics_clock>
                                <supported_graphics_clock>961 MHz</supported_graphics_clock>
                                <supported_graphics_clock>949 MHz</supported_graphics_clock>
                                <supported_graphics_clock>936 MHz</supported_graphics_clock>
                                <supported_graphics_clock>923 MHz</supported_graphics_clock>
                                <supported_graphics_clock>911 MHz</supported_graphics_clock>
                                <supported_graphics_clock>898 MHz</supported_graphics_clock>
                                <supported_graphics_clock>885 MHz</supported_graphics_clock>
                                <supported_graphics_clock>873 MHz</supported_graphics_clock>
                                <supported_graphics_clock>860 MHz</supported_graphics_clock>
                                <supported_graphics_clock>847 MHz</supported_graphics_clock>
                                <supported_graphics_clock>835 MHz</supported_graphics_clock>
                                <supported_graphics_clock>822 MHz</supported_graphics_clock>
                                <supported_graphics_clock>810 MHz</supported_graphics_clock>
                                <supported_graphics_clock>797 MHz</supported_graphics_clock>
                                <supported_graphics_clock>784 MHz</supported_graphics_clock>
                                <supported_graphics_clock>772 MHz</supported_graphics_clock>
                                <supported_graphics_clock>759 MHz</supported_graphics_clock>
                                <supported_graphics_clock>746 MHz</supported_graphics_clock>
                                <supported_graphics_clock>734 MHz</supported_graphics_clock>
                                <supported_graphics_clock>721 MHz</supported_graphics_clock>
                                <supported_graphics_clock>708 MHz</supported_graphics_clock>
                                <supported_graphics_clock>696 MHz</supported_graphics_clock>
                                <supported_graphics_clock>683 MHz</supported_graphics_clock>
                                <supported_graphics_clock>670 MHz</supported_graphics_clock>
                                <supported_graphics_clock>658 MHz</supported_graphics_clock>
                                <supported_graphics_clock>645 MHz</supported_graphics_clock>
                                <supported_graphics_clock>632 MHz</supported_graphics_clock>
                                <supported_graphics_clock>620 MHz</supported_graphics_clock>
                                <supported_graphics_clock>607 MHz</supported_graphics_clock>
                                <supported_graphics_clock>594 MHz</supported_graphics_clock>
                                <supported_graphics_clock>582 MHz</supported_graphics_clock>
                                <supported_graphics_clock>569 MHz</supported_graphics_clock>
                                <supported_graphics_clock>556 MHz</supported_graphics_clock>
                                <supported_graphics_clock>544 MHz</supported_graphics_clock>
                                <supported_graphics_clock>531 MHz</supported_graphics_clock>
                                <supported_graphics_clock>518 MHz</supported_graphics_clock>
                                <supported_graphics_clock>506 MHz</supported_graphics_clock>
                                <supported_graphics_clock>493 MHz</supported_graphics_clock>
                                <supported_graphics_clock>480 MHz</supported_graphics_clock>
                                <supported_graphics_clock>468 MHz</supported_graphics_clock>
                                <supported_graphics_clock>455 MHz</supported_graphics_clock>
                                <supported_graphics_clock>442 MHz</supported_graphics_clock>
                                <supported_graphics_clock>430 MHz</supported_graphics_clock>
                                <supported_graphics_clock>417 MHz</supported_graphics_clock>
                                <supported_graphics_clock>405 MHz</supported_graphics_clock>
                                <supported_graphics_clock>392 MHz</supported_graphics_clock>
                                <supported_graphics_clock>379 MHz</supported_graphics_clock>
                                <supported_graphics_clock>367 MHz</supported_graphics_clock>
                                <supported_graphics_clock>354 MHz</supported_graphics_clock>
                                <supported_graphics_clock>341 MHz</supported_graphics_clock>
                                <supported_graphics_clock>329 MHz</supported_graphics_clock>
                                <supported_graphics_clock>316 MHz</supported_graphics_clock>
                                <supported_graphics_clock>303 MHz</supported_graphics_clock>
                                <supported_graphics_clock>291 MHz</supported_graphics_clock>
                                <supported_graphics_clock>278 MHz</supported_graphics_clock>
                                <supported_graphics_clock>265 MHz</supported_graphics_clock>
                                <supported_graphics_clock>253 MHz</supported_graphics_clock>
                                <supported_graphics_clock>240 MHz</supported_graphics_clock>
                                <supported_graphics_clock>227 MHz</supported_graphics_clock>
                                <supported_graphics_clock>215 MHz</supported_graphics_clock>
                                <supported_graphics_clock>202 MHz</supported_graphics_clock>
                                <supported_graphics_clock>189 MHz</supported_graphics_clock>
                                <supported_graphics_clock>177 MHz</supported_graphics_clock>
                                <supported_graphics_clock>164 MHz</supported_graphics_clock>
                                <supported_graphics_clock>151 MHz</supported_graphics_clock>
                                <supported_graphics_clock>139 MHz</supported_graphics_clock>
                        </supported_mem_clock>
                        <supported_mem_clock>
                                <value>810 MHz</value>
                                <supported_graphics_clock>1746 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1733 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1721 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1708 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1695 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1683 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1670 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1657 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1645 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1632 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1620 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1607 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1594 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1582 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1569 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1556 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1544 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1531 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1518 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1506 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1493 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1480 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1468 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1455 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1442 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1430 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1417 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1404 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1392 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1379 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1366 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1354 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1341 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1328 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1316 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1303 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1290 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1278 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1265 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1252 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1240 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1227 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1215 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1202 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1189 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1177 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1164 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1151 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1139 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1126 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1113 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1101 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1088 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1075 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1063 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1050 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1037 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1025 MHz</supported_graphics_clock>
                                <supported_graphics_clock>1012 MHz</supported_graphics_clock>
                                <supported_graphics_clock>999 MHz</supported_graphics_clock>
                                <supported_graphics_clock>987 MHz</supported_graphics_clock>
                                <supported_graphics_clock>974 MHz</supported_graphics_clock>
                                <supported_graphics_clock>961 MHz</supported_graphics_clock>
                                <supported_graphics_clock>949 MHz</supported_graphics_clock>
                                <supported_graphics_clock>936 MHz</supported_graphics_clock>
                                <supported_graphics_clock>923 MHz</supported_graphics_clock>
                                <supported_graphics_clock>911 MHz</supported_graphics_clock>
                                <supported_graphics_clock>898 MHz</supported_graphics_clock>
                                <supported_graphics_clock>885 MHz</supported_graphics_clock>
                                <supported_graphics_clock>873 MHz</supported_graphics_clock>
                                <supported_graphics_clock>860 MHz</supported_graphics_clock>
                                <supported_graphics_clock>847 MHz</supported_graphics_clock>
                                <supported_graphics_clock>835 MHz</supported_graphics_clock>
                                <supported_graphics_clock>822 MHz</supported_graphics_clock>
                                <supported_graphics_clock>810 MHz</supported_graphics_clock>
                                <supported_graphics_clock>797 MHz</supported_graphics_clock>
                                <supported_graphics_clock>784 MHz</supported_graphics_clock>
                                <supported_graphics_clock>772 MHz</supported_graphics_clock>
                                <supported_graphics_clock>759 MHz</supported_graphics_clock>
                                <supported_graphics_clock>746 MHz</supported_graphics_clock>
                                <supported_graphics_clock>734 MHz</supported_graphics_clock>
                                <supported_graphics_clock>721 MHz</supported_graphics_clock>
                                <supported_graphics_clock>708 MHz</supported_graphics_clock>
                                <supported_graphics_clock>696 MHz</supported_graphics_clock>
                                <supported_graphics_clock>683 MHz</supported_graphics_clock>
                                <supported_graphics_clock>670 MHz</supported_graphics_clock>
                                <supported_graphics_clock>658 MHz</supported_graphics_clock>
                                <supported_graphics_clock>645 MHz</supported_graphics_clock>
                                <supported_graphics_clock>632 MHz</supported_graphics_clock>
                                <supported_graphics_clock>620 MHz</supported_graphics_clock>
                                <supported_graphics_clock>607 MHz</supported_graphics_clock>
                                <supported_graphics_clock>594 MHz</supported_graphics_clock>
                                <supported_graphics_clock>582 MHz</supported_graphics_clock>
                                <supported_graphics_clock>569 MHz</supported_graphics_clock>
                                <supported_graphics_clock>556 MHz</supported_graphics_clock>
                                <supported_graphics_clock>544 MHz</supported_graphics_clock>
                                <supported_graphics_clock>531 MHz</supported_graphics_clock>
                                <supported_graphics_clock>518 MHz</supported_graphics_clock>
                                <supported_graphics_clock>506 MHz</supported_graphics_clock>
                                <supported_graphics_clock>493 MHz</supported_graphics_clock>
                                <supported_graphics_clock>480 MHz</supported_graphics_clock>
                                <supported_graphics_clock>468 MHz</supported_graphics_clock>
                                <supported_graphics_clock>455 MHz</supported_graphics_clock>
                                <supported_graphics_clock>442 MHz</supported_graphics_clock>
                                <supported_graphics_clock>430 MHz</supported_graphics_clock>
                                <supported_graphics_clock>417 MHz</supported_graphics_clock>
                                <supported_graphics_clock>405 MHz</supported_graphics_clock>
                                <supported_graphics_clock>392 MHz</supported_graphics_clock>
                                <supported_graphics_clock>379 MHz</supported_graphics_clock>
                                <supported_graphics_clock>367 MHz</supported_graphics_clock>
                                <supported_graphics_clock>354 MHz</supported_graphics_clock>
                                <supported_graphics_clock>341 MHz</supported_graphics_clock>
                                <supported_graphics_clock>329 MHz</supported_graphics_clock>
                                <supported_graphics_clock>316 MHz</supported_graphics_clock>
                                <supported_graphics_clock>303 MHz</supported_graphics_clock>
                                <supported_graphics_clock>291 MHz</supported_graphics_clock>
                                <supported_graphics_clock>278 MHz</supported_graphics_clock>
                                <supported_graphics_clock>265 MHz</supported_graphics_clock>
                                <supported_graphics_clock>253 MHz</supported_graphics_clock>
                                <supported_graphics_clock>240 MHz</supported_graphics_clock>
                                <supported_graphics_clock>227 MHz</supported_graphics_clock>
                                <supported_graphics_clock>215 MHz</supported_graphics_clock>
                                <supported_graphics_clock>202 MHz</supported_graphics_clock>
                                <supported_graphics_clock>189 MHz</supported_graphics_clock>
                                <supported_graphics_clock>177 MHz</supported_graphics_clock>
                                <supported_graphics_clock>164 MHz</supported_graphics_clock>
                                <supported_graphics_clock>151 MHz</supported_graphics_clock>
                                <supported_graphics_clock>139 MHz</supported_graphics_clock>
                        </supported_mem_clock>
                        <supported_mem_clock>
                                <value>405 MHz</value>
                                <supported_graphics_clock>607 MHz</supported_graphics_clock>
                                <supported_graphics_clock>594 MHz</supported_graphics_clock>
                                <supported_graphics_clock>582 MHz</supported_graphics_clock>
                                <supported_graphics_clock>569 MHz</supported_graphics_clock>
                                <supported_graphics_clock>556 MHz</supported_graphics_clock>
                                <supported_graphics_clock>544 MHz</supported_graphics_clock>
                                <supported_graphics_clock>531 MHz</supported_graphics_clock>
                                <supported_graphics_clock>518 MHz</supported_graphics_clock>
                                <supported_graphics_clock>506 MHz</supported_graphics_clock>
                                <supported_graphics_clock>493 MHz</supported_graphics_clock>
                                <supported_graphics_clock>480 MHz</supported_graphics_clock>
                                <supported_graphics_clock>468 MHz</supported_graphics_clock>
                                <supported_graphics_clock>455 MHz</supported_graphics_clock>
                                <supported_graphics_clock>442 MHz</supported_graphics_clock>
                                <supported_graphics_clock>430 MHz</supported_graphics_clock>
                                <supported_graphics_clock>417 MHz</supported_graphics_clock>
                                <supported_graphics_clock>405 MHz</supported_graphics_clock>
                                <supported_graphics_clock>392 MHz</supported_graphics_clock>
                                <supported_graphics_clock>379 MHz</supported_graphics_clock>
                                <supported_graphics_clock>367 MHz</supported_graphics_clock>
                                <supported_graphics_clock>354 MHz</supported_graphics_clock>
                                <supported_graphics_clock>341 MHz</supported_graphics_clock>
                                <supported_graphics_clock>329 MHz</supported_graphics_clock>
                                <supported_graphics_clock>316 MHz</supported_graphics_clock>
                                <supported_graphics_clock>303 MHz</supported_graphics_clock>
                                <supported_graphics_clock>291 MHz</supported_graphics_clock>
                                <supported_graphics_clock>278 MHz</supported_graphics_clock>
                                <supported_graphics_clock>265 MHz</supported_graphics_clock>
                                <supported_graphics_clock>253 MHz</supported_graphics_clock>
                                <supported_graphics_clock>240 MHz</supported_graphics_clock>
                                <supported_graphics_clock>227 MHz</supported_graphics_clock>
                                <supported_graphics_clock>215 MHz</supported_graphics_clock>
                                <supported_graphics_clock>202 MHz</supported_graphics_clock>
                                <supported_graphics_clock>189 MHz</supported_graphics_clock>
                                <supported_graphics_clock>177 MHz</supported_graphics_clock>
                                <supported_graphics_clock>164 MHz</supported_graphics_clock>
                                <supported_graphics_clock>151 MHz</supported_graphics_clock>
                                <supported_graphics_clock>139 MHz</supported_graphics_clock>
                        </supported_mem_clock>
                </supported_clocks>
                <processes>
                </processes>
                <accounted_processes>
                </accounted_processes>
        </gpu>

</nvidia_smi_log>
root@UnRAID:~# 

 

what is populated seems to be the only things updating.  the rest stays zero and it doesn't see the streams from plex any longer.   Screenshot was during a live stream of a 4k movie.

 

Thanks @b3rs3rk

 

nvidia.png

Edited by 5STAR
  • Like 1
Link to comment

@5STAR If nvidia-smi doesn't provide me the data, I can't display it on the widget and that's exactly what I'm seeing in your output.  I'm thinking something is wrong with the NVIDIA driver's NVML library or the nvidia-smi utility itself.  You're using driver version 465.19.01 and my server is still on 460.56 and it has no such issue.  Can you try going into your Nvidia-Driver Plugin settings and rolling back the driver?  Just set it to 460.56 (which is known good for me at least) and rebooting your UnRAID server then try again.  If the widget works as expected after the rollback, I'd stay on that driver until we can determine what the issue is.  If that doesn't fix it, we'll probably have to file a bug with Nvidia or just wait for successive driver releases and hope they alleviate the issue.  I don't know, might be a better question for @ich777 as he may have heard something else about this.

image.thumb.png.19d1351309827db6b9b2f8995014957b.png
 

Link to comment
Posted (edited)
57 minutes ago, gowg said:

Love the plugin, it's working great!

 

The only thing I'd like to request is the Deepstack icon showing up for the Deepstack GPU docker. The Plex icon works

 

Request a feature enhancement on the project's Github page making sure to include the commands shown as running processes in nvidia-smi while the application is using the GPU.  Providing at least a 32x32 pixel square transparent png image to display in the widget when the app is running will make it easier for me to add as well.  Click here for a good example of such an enhancement request.

Edited by b3rs3rk
Add example link
  • Thanks 1
Link to comment
39 minutes ago, ich777 said:

@5STAR I don't have such an issue with my GTX1050ti:

grafik.png.99b935eb2f92b804c130d41a852b781c.png

 

I'm also using the newest driver version:

grafik.thumb.png.dab33991c7874d9356173e2992b882f8.png

 


It was just a stab in the dark really.  But I don't know what could be wrong with it.  My code depends on nvidia-smi functioning properly and near as I can tell my code is doing exactly what it is supposed to do.

Link to comment
1 minute ago, b3rs3rk said:


It was just a stab in the dark really.  But I don't know what could be wrong with it.  My code depends on nvidia-smi functioning properly and near as I can tell my code is doing exactly what it is supposed to do.

Eventually the card doesn't support all outputs, some drivers break the compatibility with some cards...

  • Like 1
Link to comment
1 minute ago, ich777 said:

Eventually the card doesn't support all outputs, some drivers break the compatibility with some cards...

 

Sure, but I find it hard to believe they deprecated anything with his Quadro being Pascal.

Link to comment
Just now, b3rs3rk said:

 

Sure, but I find it hard to believe they deprecated anything with his Quadro being Pascal.

Not deprecating... Think of the GTX1050Ti's, at least some of them, they work perfect about a year ago and reported everything in nvidia-smi and then after a driver update the power reading was broken, that's actually a problem with a lot of cards these days...

Link to comment
4 minutes ago, ich777 said:

Not deprecating... Think of the GTX1050Ti's, at least some of them, they work perfect about a year ago and reported everything in nvidia-smi and then after a driver update the power reading was broken, that's actually a problem with a lot of cards these days...


I just assumed they didn't have the sensors necessary to probe that data or it wasn't implemented for that GPU's driver.  Here's hoping they don't do that to the P4000 :|

Link to comment
1 minute ago, b3rs3rk said:


I just assumed they didn't have the sensors necessary to probe that data or it wasn't implemented for that GPU's driver.  Here's hoping they don't do that to the P4000 :|

The sensors are working in Windows for example, the problem is that some newer features break things for other "older" cards in nvidia-smi and that's it... I think you know what I mean... Nvidia...

Link to comment

Dang this was working great until the flurry of 5 or so upgrades in a week about 3 weeks ago.   I rolled all the way back to v455.45.01 and it is still broke in the same areas.   

 

Wonder what changed...

Link to comment
47 minutes ago, 5STAR said:

Dang this was working great until the flurry of 5 or so upgrades in a week about 3 weeks ago.   I rolled all the way back to v455.45.01 and it is still broke in the same areas.

Then it has to be something different even if the old drivers aren't working...

Eventually it's hardware related.

  • Like 1
Link to comment
Posted (edited)
2 hours ago, 5STAR said:

Dang this was working great until the flurry of 5 or so upgrades in a week about 3 weeks ago.   I rolled all the way back to v455.45.01 and it is still broke in the same areas.   

 

Wonder what changed...


You're free to roll back to a previous version of my plugin by manually installing the PKG file from the Github repository if you're insistent that it was from my code changes.   All of the files are dated, so you can go back as far as you want.  But if your nvidia-smi continues to not display the metrics that aren't updating, it's not going to make any difference as that's where I get the data from.

EDIT:  After reviewing your setup in your signature block, I don't think anyone can help you unless someone reports a similar problem that has a less complex installation of UnRAID running.  To be frank, there are plenty of things that can go wrong or function differently if you're running UnRAID within a VM, especially when attempting to pass through a GPU to it and poll the statistics as if it were bare metal.  I would reach out to the folks in the sub forum dedicated to running UnRAID as a VM and see if anyone is having a similar issue and how to resolve it.

Edited by b3rs3rk
Link to comment
3 hours ago, b3rs3rk said:


You're free to roll back to a previous version of my plugin by manually installing the PKG file from the Github repository if you're insistent that it was from my code changes.   All of the files are dated, so you can go back as far as you want.  But if your nvidia-smi continues to not display the metrics that aren't updating, it's not going to make any difference as that's where I get the data from.

EDIT:  After reviewing your setup in your signature block, I don't think anyone can help you unless someone reports a similar problem that has a less complex installation of UnRAID running.  To be frank, there are plenty of things that can go wrong or function differently if you're running UnRAID within a VM, especially when attempting to pass through a GPU to it and poll the statistics as if it were bare metal.  I would reach out to the folks in the sub forum dedicated to running UnRAID as a VM and see if anyone is having a similar issue and how to resolve it.

I notice the supported memory clock isn't in the list:

 <supported_mem_clock>
                                <value>5005 MHz</value>
                                <supported_graphics_clock>1746 MHz</supported_graphics_clock>

Link to comment
4 minutes ago, 5STAR said:

I notice the supported memory clock isn't in the list:

 <supported_mem_clock>
                                <value>5005 MHz</value>
                                <supported_graphics_clock>1746 MHz</supported_graphics_clock>

 

I don't use any of the data in the <supported_clocks> parent element.  It could be entirely non-existent from the nvidia-smi output and it wouldn't make any difference.  So whatever you're seeing is probably irrelevant.

Link to comment

I think this is an api problem with nvidia.  I can validate that the hw transcoding is working in plex as I see the hw next to the stream.   The quadro p2200 is seen by everything including unraid and the plex docker container.  I can launch the plex docker terminal I can run nvidia-smi and it works there too.  So I know the hardware and passthrough is working great to the docker container.   The variables that are seen in nvidia-smi do update on the gpustats plugin.  it's just not passing all it used to pass right now with null data for things like utilization, encode/decode %'s etc.. 

 

Like I said this busted in one of the multiple updates a week or 2 ago.   There were several nvidia updates as well as gpustats.  This may be nvidia driver issues with api. 

 

smi.png

 

I added the nvidia-smi showing the process starting for transcoder but most of the data missing. 

 

I think there are a separate series of drivers for quadro than geforce or rtx drivers now

 

@b3rs3rk @ich777

Edited by 5STAR
Link to comment
11 hours ago, 5STAR said:

I think this is an api problem with nvidia.  I can validate that the hw transcoding is working in plex as I see the hw next to the stream.   The quadro p2200 is seen by everything including unraid and the plex docker container.  I can launch the plex docker terminal I can run nvidia-smi and it works there too.  So I know the hardware and passthrough is working great to the docker container.   The variables that are seen in nvidia-smi do update on the gpustats plugin.  it's just not passing all it used to pass right now with null data for things like utilization, encode/decode %'s etc.. 

 

Like I said this busted in one of the multiple updates a week or 2 ago.   There were several nvidia updates as well as gpustats.  This may be nvidia driver issues with api. 

 

I added the nvidia-smi showing the process starting for transcoder but most of the data missing. 

 

I think there are a separate series of drivers for quadro than geforce or rtx drivers now

 

@b3rs3rk @ich777


I updated to 6.9.2 today and updated to the latest NVIDIA driver (465.19.01) and I'm still not having any issues with a Quadro P4000.  I've already pushed a fix for the double NVIDIA in the product name which will go in the next release.  Other than that, the plugin is displaying data just as I expected.

image.png.05da8436e6940779ed9b5a1aa2e50759.png

Link to comment
On 4/6/2021 at 4:17 PM, b3rs3rk said:

 

Request a feature enhancement on the project's Github page making sure to include the commands shown as running processes in nvidia-smi while the application is using the GPU.  Providing at least a 32x32 pixel square transparent png image to display in the widget when the app is running will make it easier for me to add as well.  Click here for a good example of such an enhancement request.

nvidia-smi just says "python3" so I'm assuming that means it's impossible to track Deepstack?

Link to comment
8 minutes ago, gowg said:

nvidia-smi just says "python3" so I'm assuming that means it's impossible to track Deepstack?

 

Not necessarily.  If the full command invocation contains keywords that tie the process to deepstack I can add that as a detection with recent code changes.  When deepstack is using the GPU, get the PID from nvidia-smi and then run:
 

echo $(ps -fp <insert_pid_from_nvidia-smi> -o command)


And paste me the full result.
 

Link to comment
3 minutes ago, b3rs3rk said:

 

Not necessarily.  If the full command invocation contains keywords that tie the process to deepstack I can add that as a detection with recent code changes.  When deepstack is using the GPU, get the PID from nvidia-smi and then run:
 


echo $(ps -fp <insert_pid_from_nvidia-smi> -o command)


And paste me the full result.
 

COMMAND python3 /app/intelligencelayer/shared/face.py

COMMAND python3 /app/intelligencelayer/shared/detection.py
 

there's 2 processes I guess

Link to comment
Posted (edited)

@gowg We might have to do something different with this one.  Using the same PID from nvidia-smi get the parent PID:

 

ps -o ppid= -p <insert_nvidia-smi_pid>

 

Then take that pid and feed it into the previous command I sent you instead of the PID from nvidia-smi and paste me the result.  I'm hoping that the parent thread that spawns these python3 processes has obvious language involving deepstack.

Edited by b3rs3rk
Link to comment
2 hours ago, b3rs3rk said:

@gowg We might have to do something different with this one.  Using the same PID from nvidia-smi get the parent PID:

 


ps -o ppid= -p <insert_nvidia-smi_pid>

 

Then take that pid and feed it into the previous command I sent you instead of the PID from nvidia-smi and paste me the result.  I'm hoping that the parent thread that spawns these python3 processes has obvious language involving deepstack.

COMMAND python3 /app/intelligencelayer/shared/detection.py

COMMAND python3 /app/intelligencelayer/shared/face.py

 

shucks, seems to be the same

Link to comment
Just now, gowg said:

COMMAND python3 /app/intelligencelayer/shared/detection.py

COMMAND python3 /app/intelligencelayer/shared/face.py

 

shucks, seems to be the same


Just to check, the parent PID wasn't the same as the PID in nvidia-smi right?  I may have something wrong in my thinking here.  Ugh, makes it so much harder when I can't test this stuff on my own.  I'll see if I can run a deepstack instance and get it to spawn these processes.  I'm guessing you need a camera to do it?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.