Android 5.1 encryption password gone after changing back to pattern

I've encrypted my phone (Android 5.1) after setting up a password, and it worked fine.

Because I don't want to use a password for the normal screen unlock, I've changed it back to pattern.

But now when I restart the phone, I'm not asked for the password anymore. I remember I've set this up before, password to boot, and pattern to unlock screen.

What did I do wrong?