Just tried it here (with the trial version too) and, at least the last release (v2.6.3), seems to be working just fine. Tried exactly the command line you posted above. The "entry missing" error will indeed occur if you try to call it from the 64-bit dll, PDFShellTools64.dll, as the functions are only implemented in the 32-bit dll.
By the way, current versions don't need that rundll32 calling schema (I need to update the user's guide). You may just call the more handy command line tool, pdfshelltools.exe, instead, i.e.
"C:\Program Files (x86)\PDF-ShellTools\PDFShellTools.exe" SetResetSecurity PermPass=MonMotDePasse EncLevel=2 -CreateBackup c:\test\a.pdf