Reverse engineering and fine-tuning Asustor NAS fans

By Ricardo

November 14, 2021

Background story

Recently Ewen Bell contacted me on Twitter regarding my posts about the Asustor NAS and the reverse engineering progress on it. The whole thread is worth reading, actually.

What happened to the blog posts about the Asustor? I found it in Google cache while trying to find a way to set more effective temperature triggers for the fan.
— ‏Ewen Bell (@ewster) November 3, 2021

Btw, if you want, there’s a very nice review also from Ewen in his blog. You definitely want to check it if you want the whole story on why was I so interested in this project! :)

Anyway, the main question is: can we control the fan better than the original firmware? And if so, how? You see, the main firmware only allows 4 fan speed settings: low, medium, high or auto. Being so limit is annoying - plus if you ever need to change the “auto” speed curve manually, you just are not able to. That sucks, Asustor.

Let’s do it!

Ok, so from the files in the NAS itself and previous experience looking at this topic, we know that the fan speed is something configured (or at least once was configured) in the /etc/nas.conf:

# cat /etc/nas.conf  | grep -i fan
FanNumber = 1
IsSmartFan = Yes
FanSpeed = High

Not many options...

These strings are useful, and the IsSmartFan and FanSpeed strings seem very promising. We can now look at the firmware to find any file (binary or script!) that references those:

$ grep -r IsSmartFan *
initramfs_unpacked/etc/default/nas.conf:IsSmartFan = Yes

No such luck on this one it seems

$ grep -r FanSpeed *
Binary file initramfs_unpacked/usr/sbin/emboardmand matches
initramfs_unpacked/etc/default/nas.conf:FanSpeed = High

This seems a bit more promising!

So we meet the daemons again! sigh ok, let’s dive into it then.

And deeper we go!

After setting it up on Ghidra, the next step was to look for any occurrence of the FanSpeed string, which gave a single entry: "[EMBOARD] %s(%d):\t --> iFixedPwm %d, idFanSpeed %d\n". There’s only a single reference to that string, in the unnamed function FUN_0040bfc0. Its usage seems to be as a debug string, which makes sense:

if (DAT_006159d5 != '\0') {
    printf("[EMBOARD] %s(%d):\t --> iFixedPwm %d, idFanSpeed %d\n", "Fan_Service_Handler_7XXT",0x5f7);
}

That DAT_006159d5 is read in many places before a prinntf, so we’ll just rename it to DEBUG_FLAG to keep it clean. The Fan_Service_Handler_7XXT is used in all debug strings in this functions, and based on the other ones this function is actually the fan service handler. Good, we seem to be in the right track then, I guess? This function also calls a very interesting one: Hal_Fan_Set_Raw_Value, imported from an external library, libnhal.so. Opening that lib gives us some very interesting exports!

Cool! And since we know how to call such exports (kinda), we can test them!

The first test was seeing what Hal_Fan_Get_Raw_Value returns. Unfortunately, it always returned 0 for me, so I decided to skip that one.

Ricardo from the future here! Actually, I was calling Hal_Fan_Get_Raw_Value wrong - it was missing the second parameter! Once I added that, I was able to read the raw PWM value set from the daemon. Nice!

Next interesting one is Hal_Fan_Get_Speed, which simply reads the fan speed (in RPM):

#include <stdio.h>
#include <stdlib.h>

extern int Hal_Fan_Get_Speed(uint param_1,int *param_2);

int main() {
  int speed = 0;
  int retval = Hal_Fan_Get_Speed(0, &speed);
  printf("speed = %d, retval = %d\n", speed, retval);

  return 0;
}

# ./fan1
speed = 526, retval = 0

"Low" is ~520 RPM, so sounds about right.

Ok, nice, now we have a tool to get the fan speed. However, we don’t want that - we want a tool to set fan speed. But there’s no Hal_Fan_Set_Speed function :(

However, if we go back to that fan service handler function, there are calls to a Hal_Fan_Set_Raw_Value… could that be it? There are two calls to it:

// (...)
ivar3 = Hal_Fan_Set_Raw_Value((ulong)uVar9,(ulong)((uint)uVar10 & 0xff));
// (...)
ivar3 = Hal_Fan_Set_Raw_Value((ulong)uVar9,(ulong)local_a8);
// (...)

It can't be that easy... can it?

So I decided to test it: what happens if I call that function with my fan index (which is zero btw) and… let’s say, 255, as it seems to be masked with 0xff? Well… this happens:

#include <stdio.h>
#include <stdlib.h>

extern int Hal_Fan_Set_Raw_Value(ulong p1, uint p2);

int main(int argc, char *argv[]) {
  if (argc != 3) {
    printf("%s <fan index> <pwm value (0-255)>\n", argv[0]);
    return 1;
  }

  int fan_index = atoi(argv[1]);
  uint fan_pwm = atoi(argv[2]) & 0xff;

  printf("Setting fan %d PWM value to %u\n", fan_index, fan_pwm);
  int retval = Hal_Fan_Set_Raw_Value(fan_index, fan_pwm);
  if (retval != 0) {
    printf("FAILED: return value is %d\n", retval);
    return 1;
  }

  return 0;
}

# ./fan3
./fan3 <fan index> <pwm value (0-255)>

# ./fan3 0 255
Setting fan 0 PWM value to 255

# ./fan1
speed = 2227, retval = 0

# ./fan3 0 0
Setting fan 0 PWM value to 0

# ./fan1
speed = 0, retval = 0

fan1 reads the speed, fan3 sets the speed

Let’s just say it got loud! Yey, success!

There’s something “wrong” though: as soon as I set the fan speed, it’ll soon go down to the original value. This is not actually a bug in my code (ironically), but a feature from the NAS itself: it keeps reading whatever setting is in the configuration and changes the PWM to it. And guess who is doing that? Yes, emboardmand, that pesky daemon!

Daemons should stay in hell

I can’t stop emboardmand - or at least I don’t want to. You see, this daemon exists for a reason, and since I don’t own the original source code or even begin fully understand what it is doing, I strongly believe killing or suspending its process is a bad idea. It would stop the fan service and allow us to control it by ourselves, but at what cost? Yeah, no, we need a better solution.

Ricardo from the future here: the daemon seems to control other things indeed. Based on a few basic checks, it controls the LEDs, power, buttons, hibernation and system checks. Definitely something you don’t want to stop!

By looking at the code (which I won’t post here as it’s very long and not fully reverse engineered) I can see it runs on a thread, which has an infinite loop (with a break condition) that runs its code every 10 seconds. The frequency is controlled by a sleep call with the 10 constant, hence the 10s. I could try to force the thread to get suspended, but as far I’ve been reading on the Linux thread management and POSIX threads, that is not possible. I could try to get the thread to exit by changing the value of its break condition, but that seems very dangerous due to the fact we would be both changing a global variable in the daemon (which could be used to control other threads!) and actually changing a process memory, which is a bad idea.

Let’s study some ideas and check whether they are any good. I’ll take you through my through process, so enjoy the ride!

First idea

How about we take a very easy approach here? Let’s say we read the current speed or PWM value, compare to the our expected values and, if they don’t match, update the PWM? Basically brute-force the damn thing.

Pros:

No system change whatsoever! We just need to run a small application inside the system and that’s it.

Cons:

Noise. If we take too long to ramp up or down the fans, we would get noise from the system changing its speed.
Fan degradation. I’m pretty sure spinning up and down a fan would probably cause some degradation of it. Plus, I think it might increase power consumption due to the continuous slow down followed by the spin up.
Fan lock up. Yeah, I tried setting the fan speed every 100ms and if I stopped the process mid-change, it would lock up the fan control. Good for stopping the daemon, bad because we can’t change the damn thing now.

Second idea

How about we pass modified libraries to the daemon? We’ve been doing this for a while in the previous experiments with the NAS firmware image, could it be done in the real one?

Pros:

It’s a permanent solution, as we would take full control of the fan.

Cons:

If we fuck it up, the system has no fan control, which could lead to overheat and system damage.
Hard to apply as it requires modifying system files to make it work. Also, if for some reason the daemon fails to start (let’s say due to a change in the libraries), the whole OS will reboot after a few seconds as we’ve seen in our previous experimentation.

Third idea

Take control only of the semaphore locking the fan speed change. Based on the Hal_Fan_Set_Raw_Value code, we can see it calls different functions for different platforms, as the firmware is generic up to this point.

int Hal_Fan_Set_Raw_Value(int fan_index,int fan_pwm)

{
  int iVar1;
  long lVar2;
  int local_20;
  int local_1c [3];

  local_20 = 0;
  local_1c[0] = 0;
  iVar1 = Hal_Get_Platform_Fan_Count(local_1c);
  if (-1 < iVar1) {
    iVar1 = -0x16;
    if ((-1 < fan_index) && (fan_index <= local_1c[0])) {
      lVar2 = Hal_Get_Platform_Identity(&local_20);
      iVar1 = (int)lVar2;
      if (-1 < iVar1) {
        if (local_20 == 1) {
LAB_001083d0:
          iVar1 = It87_Set_Fan_Pwm(fan_index,fan_pwm & 0xff);
          return iVar1;
        }
        if (local_20 == 2) {
          iVar1 = Lm63_Set_Fan((ulong)(uint)fan_index,(ulong)(uint)fan_pwm & 0xff);
        }
        else {
          if (local_20 - 0x15U < 7) goto LAB_001083d0;
          if (local_20 - 3U < 2) {
            iVar1 = Mcu_Set_Fan_Pwm((ulong)(uint)fan_pwm & 0xff);
          }
        }
      }
    }
  }
  return iVar1;
}

Fancy!

In my model, I belived the called function is the It87_Set_Fan_Pwm, which is defined in the libgeneraldrv. This function, however, is controlled by a semaphore, as it can be seen in the code. Also, this explains why I managed to lock the fan control up in the first idea. Oops!

ulong It87_Set_Fan_Pwm(int param_1,int param_2)
{
  uint uVar1;
  uint uVar2;
  ulong uVar3;
  long lVar4;
  undefined8 extraout_RDX;
  undefined8 *puVar5;
  uint uVar6;
  int iVar7;
  undefined8 auStack104 [7];

  uVar3 = 0xffffffea;
  if ((uint)param_1 < 3) {
    lVar4 = 6;
    puVar5 = auStack104;
    while (lVar4 != 0) {
      lVar4 = lVar4 + -1;
      *puVar5 = 0;
      puVar5 = puVar5 + 1;
    }
    uVar3 = Hw_Semaphore_Init(auStack104,"/HW_SEMAPHORE_IT87XX",1);
    if (-1 < (int)uVar3) {
      uVar3 = Hw_Semaphore_Lock(auStack104);
 
 // (...)

Red light to anything messing up with IT87!

Pros:

Full control of the fan, as we can control the semaphore. We just lock it up and control the way we want.
No changes of any kind to the original OS.

Cons:

Risk of getting the semaphore locked forever. We have to handle every single error to avoid our process fucking up the semaphore. I would go as far as have a separate process monitoring the new fan service and, whenever it dies, releases control to the OS.
Very, very platform dependent. This would work on any IT87-based model (most likely Intel-based ones), but this would not be good for, you know, the community.
Asking the weird questions: if I take control of the semaphore and the function tries do to the same, does it work or does it just halts?

Fourth idea

Let’s use the CGI and keep the speed at low/medium/high based on custom values. The web interface calls a CGI to change the fan speed, and we could do the same. We can easily force the call behind it by either reverse engineering the CGI binary or simply fake the call with some modified libraries to accepted unauthenticated requests from the terminal, as we’ve done in the past.

Pros:

Does not mess with the fans PWM directly, but instead uses the original NAS values.
Custom program to control this is very easy to implement.

Cons:

If something goes wrong, the NAS could be stuck at low fan speed during intensive operations, which could lead to overheating and damage!
No fine-tunning of the fan speed.

So, what should I do?

Designing a solution

Ok, based on all the ideas we’ve talked so far, we need to design a solution that meets the following requirements:

Must not require any OS or firmware modification.
Must not crash the OS (or at least tries not to) if something goes wrong.
Must not use 100% CPU (no infinite loops without sleeps!).
Must not lock up the fan speed in something too low if something goes wrong.
Must be platform-independent, meaning that I don’t need to know which fan system are you using behind this thing.
Must allow the fan curve to be configurable, meaning it should be able to set the fan speed based on the temperatures I decide.
Must be generic regarding the number of fans: if you have 10 of them, I want to be able to control them all.
Must be generic regarding the number of disks: if you have 4 or 10 disks, it doesn’t matter, all of them will be used to decide the new speed.

Based on this, I’ve decided to go with the first idea: read the current PWM value, compare, and if it’s different, update it. However, this time I’m handling some signals so I can control the interruption of the program to make sure it will stop the loop (and therefore release the semaphores) and only then exit. I’m also controlling the speed of the loop to make sure it runs without stressing the CPU too much. Not only that, I’m making everything a bit more dynamic and “Asustor-style”, such as reading the temperatures using their own functions.

And, believe it or not, this is what I got:

Yes, it’s real! I’ve published a repository for this and other future Asustor hacking tools! 🎉🎉🎉

The fan-control tool allows you to fine-tune the fan curve to the way you like. It requires only 3 arguments:

The fans you want to change (their indexes in the OS)
The amount of time in nanoseconds you want to sleep between each check
The fan curve

This small program works by running a “breakable infinite loop” and checking on each cycle the fans PWM value. If for some reason they don’t match what we expected, it updates it. We’re basically competing with emboardmand on who controls the fan speed faster! Is it ugly? Yes. Does it work? Yes!

The curve is defined by specifying the base temperature and the PWM value for it. For example, the string 30:0,40:50,50:100,60:150,70:255 means the following:

39 °C and below: 0% (PWM 0)
40-49 °C: ~19% (PWM 50)
50-59 °C: ~39% (PWM 100)
60-69 °C: ~58% (PWM 150)
70 °C and above: 100% (PWM 255)

Or, even better, let’s show the curve:

Ricardo from the future here again! Please do not run your NAS at 0% fan speed. It can cause the CPU overheat and get damage or damage nearby components, besides obviously causing crashes. Make sure your first entry in the curve has more than 0% speed. Thank you.

That simple. This should help Ewen on his issue, even though it’s not an easy install yet: you still need to run this on a screen session or something like that as I’m too lazy for coding a daemon or writing a proper Asustor application package and service scripts. Ah, and if it crashes (because I’m an idiot and I’m not checking all return values), nothing bad should happen, as we’ll just stop changing the PWM and emboardmand will take over.

This code requires firmware version 4.0.0.RN53 or higher and was tested only on my own NAS, an AS3104T. The firmware requirement is because at version 3.4 there was no easy way to get the HDD temperature from the HAL interface, but I didn’t check any other firmware versions. Your mileage may vary.

Obviously you should run this code at your own risk. If something gets too hot and burns down your NAS, I’m not responsible. You should always read the code before building and running it. Also, please, have backups!

See you all the next time! :)

Reverse Engineering