Extract file Checksum

[Padanges]

Hi,
it would be useful to add Checksum values (MD5, SHA-1) to DB for each pdf file. Is there any way to accomplish that using PDFe version 1.55.6.2? If no, would you consider including that function?

Regards.

[RTT]

Try with this vbscript script function. To be used the same way as explained in your other post about existence of attachments. This function must take a parameter to specify the hash algorithm.
This are the algorithms supported:
[CalcHash](md5)
[CalcHash](sha1)
[CalcHash](sha256)
[CalcHash](sha384)
[CalcHash](sha512)
[CalcHash](ripemd160)

Code: [Select]

Function CalcHash()
 Select Case CurrentField.value
  Case "md5"        CalcHash=md5(BatchFile.filename)
  Case "sha1"       CalcHash=sha1(BatchFile.filename)
  Case "sha256"     CalcHash=sha256(BatchFile.filename)
  Case "sha384"     CalcHash=sha384(BatchFile.filename)
  Case "sha512"     CalcHash=sha512(BatchFile.filename)
  Case "ripemd160"  CalcHash=ripemd160(BatchFile.filename)       
  Case Else CalcHash=""
 End Select
End Function

'from http://mwganson.freeyellow.com/md5/
Function md5(filename)
Dim MSXML, EL, MD5Obj

    Set MD5Obj = CreateObject("System.Security.Cryptography.MD5CryptoServiceProvider")
MD5Obj.ComputeHash_2(readBinaryFile(filename))

Set MSXML = CreateObject("MSXML2.DOMDocument")
Set EL = MSXML.CreateElement("tmp")
EL.DataType = "bin.hex"
EL.NodeTypedValue = MD5Obj.Hash
md5 = EL.Text
End Function

Function sha1(filename)
Dim MSXML, EL
Set SHA1 = CreateObject("System.Security.Cryptography.SHA1CryptoServiceProvider")
SHA1.ComputeHash_2(readBinaryFile(filename))

Set MSXML = CreateObject("MSXML2.DOMDocument")
Set EL = MSXML.CreateElement("tmp")
EL.DataType = "bin.hex"
EL.NodeTypedValue = SHA1.Hash
sha1 = EL.Text
End Function

Function sha256(filename)
Dim MSXML, EL
Set SHA256 = CreateObject("System.Security.Cryptography.SHA256Managed")
SHA256.ComputeHash_2(readBinaryFile(filename))

Set MSXML = CreateObject("MSXML2.DOMDocument")
Set EL = MSXML.CreateElement("tmp")
EL.DataType = "bin.hex"
EL.NodeTypedValue = SHA256.Hash
sha256 = EL.Text
End Function

Function sha384(filename)
Dim MSXML, EL
Set SHA384 = CreateObject("System.Security.Cryptography.SHA384Managed")
SHA384.ComputeHash_2(readBinaryFile(filename))

Set MSXML = CreateObject("MSXML2.DOMDocument")
Set EL = MSXML.CreateElement("tmp")
EL.DataType = "bin.hex"
EL.NodeTypedValue = SHA384.Hash
sha384 = EL.Text
End Function


Function sha512(filename)
Dim MSXML, EL
Set SHA512 = CreateObject("System.Security.Cryptography.SHA512Managed")
SHA512.ComputeHash_2(readBinaryFile(filename))

Set MSXML = CreateObject("MSXML2.DOMDocument")
Set EL = MSXML.CreateElement("tmp")
EL.DataType = "bin.hex"
EL.NodeTypedValue = SHA512.Hash
sha512 = EL.Text
End Function

Function ripemd160(filename)
Dim MSXML, EL
Set ripemd160 = CreateObject("System.Security.Cryptography.ripemd160Managed")
ripemd160.ComputeHash_2(readBinaryFile(filename))

Set MSXML = CreateObject("MSXML2.DOMDocument")
Set EL = MSXML.CreateElement("tmp")
EL.DataType = "bin.hex"
EL.NodeTypedValue = ripemd160.Hash
ripemd160 = EL.Text
End Function

Function readBinaryFile(filename)
Const adTypeBinary = 1
Dim objStream
Set objStream = CreateObject("ADODB.Stream")
objStream.Type = adTypeBinary
objStream.Open
If filename <> "" Then objStream.LoadFromFile filename End If 'slight modification here to prevent error msg if no file selected
readBinaryFile = objStream.Read
objStream.Close
Set objStream = Nothing
End Function


[Padanges]

Thanks. The code works fine.
Is it possible to make equivalent checks/operations using jScript, and not vbScript?

[RTT]

You may create the above code as a script in the libraries section of the scripts editor. Then you may call any of its functions from any script, even from a different language, by including its reference (i.e. checking its name in the list of includes) in that script.

In the "libraries" tab create a VBScript and call it HashUtils. Put there the above code, but replace the CalcHash function with this one:

Code: [Select]

Function GetFileHash(filename,hashtype)
 Select Case hashtype
  Case "md5"        GetFileHash=md5(filename)
  Case "sha1"       GetFileHash=sha1(filename)
  Case "sha256"     GetFileHash=sha256(filename)
  Case "sha384"     GetFileHash=sha384(filename)
  Case "sha512"     GetFileHash=sha512(filename)
  Case "ripemd160"  GetFileHash=ripemd160(filename)       
  Case Else GetFileHash=""
 End Select
End Function

Now, in the "scripts" tab, create a JScript named CalcHash with this code:

Code: [Select]

function CalcHash() {
    return GetFileHash(BatchFile.filename, CurrentField.value);
}
To reference the HashUtils library, that contains the implementation of GetFileHash function, click the includes button, the button at the right of the script language selector (check the second screenshot of this forum post for reference), and check the HashUtils entry. Now you have a jscript that is using functionalities from a VBScript.

Take notice that I've just found a bug while testing this. You will need to close the batch tool, and reopen it, when creating/making changes to a script in the libraries, or the called script function will not work immediately after closing the editor.

[Padanges]

Actually, each time I change code in any Script, it does not work immediately after closing the editor.
Strangely enough, when I run debug for the code on the Jscript side, it returns me a number as a command prompt answer. But the Edit Info batch tool returns only empty strings (even when I change return to "any text" on Case Esle in VBscript). Something's buggy here.

[Padanges]

Moreover, I found (using the first version of the code) that large files fail to get any checksum at all. It's usually for files over 100mb, though on some occasion even they sometimes get the calculations done. The 600mb file - failed always. But that was not a problem for external software tools.
Could you investigate this issues? Why is this script not fully working? Memory limitations? How to overrun them? How to estimate limits?

[RTT]

Actually, each time I change code in any Script, it does not work immediately after closing the editor.
Strangely enough, when I run debug for the code on the Jscript side, it returns me a number as a command prompt answer. But the Edit Info batch tool returns only empty strings (even when I change return to "any text" on Case Esle in VBscript). Something's buggy here.

I'm testing with my development version and I forgot that the version you have doesn't support library includes in these tools (Renamer, Edit Info Fields, etc.) that call a specif function in the script. It works for the My Scripts batch tool, and from the script editor, but only after that 1.5.66.2 version got release I fixed this problem in these other tools. My other tool, the PDF-ShellTools, have this already fixed, so you may test this with it.

Moreover, I found (using the first version of the code) that large files fail to get any checksum at all. It's usually for files over 100mb, though on some occasion even they sometimes get the calculations done. The 600mb file - failed always. But that was not a problem for external software tools.
Could you investigate this issues? Why is this script not fully working? Memory limitations? How to overrun them? How to estimate limits?

The problem with that script is that it loads all the file content into memory and so it fails with these bigger files. Anyway, here is a totally rewritten CalcHash function, now in JScript language, that calculates the file hash in chunks. Take note the main function name is CalcHashJS, so name your script, and call tags, accordingly. For convenience (for other, not so experienced users), I'm including the stand alone myscript file attached, so you just need to import it.

Code: [Select]

var EL;

function CalcHashJS() {
    if (!EL) {
        var MSXML = new ActiveXObject("MSXML2.DOMDocument");
        EL = MSXML.createElement("tmp");
        EL.dataType = "bin.hex";
    }
    EL.nodeTypedValue = GetFileHash(BatchFile.Filename, CurrentField.Value);
    return EL.text;
}

function GetFileHash(filename, hashtype) {
    var stream = GetBinaryFileStream(filename);
    try {
        switch (hashtype) {
        case "md5":
            return ComputeHash(stream, "System.Security.Cryptography.MD5CryptoServiceProvider");
            break;
        case "sha1":
            return ComputeHash(stream, "System.Security.Cryptography.SHA1Managed");
            break;
        case "sha256":
            return ComputeHash(stream, "System.Security.Cryptography.SHA256Managed");
            break;
        case "sha384":
            return ComputeHash(stream, "System.Security.Cryptography.SHA384Managed");
            break;
        case "sha512":
            return ComputeHash(stream, "System.Security.Cryptography.SHA512Managed");
            break;
        case "ripemd160":
            return ComputeHash(stream, "System.Security.Cryptography.ripemd160Managed")
            break;
        default:
            return "";
        }
    } catch (err) {
        return err.message
    } finally {
        stream.Close;
    }
}

function ComputeHash(stream, objectid) {
    var BlockSize = 65536;
    var HashObj = new ActiveXObject(objectid);
    try {
        var nBlocks = parseInt(stream.size / BlockSize);
        for (var i = 1; i < nBlocks; i++) {
            var block = stream.read(BlockSize);
            HashObj.TransformBlock(block, 0, BlockSize, block, 0);
        }
        var n = stream.size - stream.position;
        block = stream.read(n);
        HashObj.TransformFinalBlock(block, 0, n);
        return HashObj.Hash;
    } finally {
        HashObj.Clear
    }
}

function GetBinaryFileStream(filename) {
    objStream = new ActiveXObject("ADODB.Stream");
    objStream.Type = 1; //adTypeBinary
    objStream.Open();
    objStream.LoadFromFile(filename);
    return objStream
}


[Padanges]

Thanks! But there's still one problem - we need to fix the value return for the archived files: they get extracted, and get log status "OK" but the value is not returned for the entry.

By the way, what do you get when you write "var EL;" before the "function CalcHashJS() {" line? The code works fine with the line under the begin/end function scope. And what does EL stand for? Just curious

[RTT]

Thanks! But there's still one problem - we need to fix the value return for the archived files: they get extracted, and get log status "OK" but the value is not returned for the entry.

Now with code to deal with archived files:

Code: [Select]

var fso = new ActiveXObject("Scripting.FileSystemObject");
var EL;

function CalcHashJS() {
    if (!EL) {
        var MSXML = new ActiveXObject("MSXML2.DOMDocument");
        EL = MSXML.createElement("tmp");
        EL.dataType = "bin.hex";
    }

    var filename = BatchFile.Filename;
    if (filename.indexOf('>') > 0) { //file is archived
        filename = fso.getFileName(filename); //get archived file name, by removig the archive name and any internal path
        filename = fso.BuildPath(fso.GetSpecialFolder(2 /*TemporaryFolder*/ ), filename);
        BatchFile.BookmarkRoot; //trick to trigger the file extraction from the archive to the system temporary folder
        if (!fso.FileExists(filename)) return "extraction from archive failed"
    }

    EL.nodeTypedValue = GetFileHash(filename, CurrentField.Value);
    return EL.text;
}

function GetFileHash(filename, hashtype) {
    var stream = GetBinaryFileStream(filename);
    try {
        switch (hashtype) {
        case "md5":
            return ComputeHash(stream, "System.Security.Cryptography.MD5CryptoServiceProvider");
            break;
        case "sha1":
            return ComputeHash(stream, "System.Security.Cryptography.SHA1Managed");
            break;
        case "sha256":
            return ComputeHash(stream, "System.Security.Cryptography.SHA256Managed");
            break;
        case "sha384":
            return ComputeHash(stream, "System.Security.Cryptography.SHA384Managed");
            break;
        case "sha512":
            return ComputeHash(stream, "System.Security.Cryptography.SHA512Managed");
            break;
        case "ripemd160":
            return ComputeHash(stream, "System.Security.Cryptography.ripemd160Managed")
            break;
        default:
            return "";
        }
    } catch (err) {
        return err.message
    } finally {
        stream.Close;
    }
}

function ComputeHash(stream, objectid) {
    var BlockSize = 65536;
    var HashObj = new ActiveXObject(objectid);
    try {
        var nBlocks = parseInt(stream.size / BlockSize);
        for (var i = 1; i < nBlocks; i++) {
            var block = stream.read(BlockSize);
            HashObj.TransformBlock(block, 0, BlockSize, block, 0);
        }
        var n = stream.size - stream.position;
        block = stream.read(n);
        HashObj.TransformFinalBlock(block, 0, n);
        return HashObj.Hash;
    } finally {
        HashObj.Clear
    }
}

function GetBinaryFileStream(filename) {
    objStream = new ActiveXObject("ADODB.Stream");
    objStream.Type = 1; //adTypeBinary
    objStream.Open();
    objStream.LoadFromFile(filename);
    return objStream
}


By the way, what do you get when you write "var EL;" before the "function CalcHashJS() {" line? The code works fine with the line under the begin/end function scope. And what does EL stand for? Just curious

Just a small optimization, and there are other objects (e.g. the stream) that can have similar optimization. When used by the tools, on multiple files, this function will be called multiple times too, so no need to create the MSXML and add the node element (the "EL", used to convert the variant array of bytes hash to hexadecimal representation) with each call. Just put it on a global variable and reuse it on the
next calls.
Replace the above script "CalcHashJS" function with this one:

Code: [Select]

function CalcHashJS() {
    var msg = '';
    if (!EL) {
        var MSXML = new ActiveXObject("MSXML2.DOMDocument");
        EL = MSXML.createElement("tmp");
        EL.dataType = "bin.hex";
        msg = '(EL created)';
    }

    var filename = BatchFile.Filename;
    if (filename.indexOf('>') > 0) { //file is archived
        filename = fso.getFileName(filename); //get archived file name, by removig the archive name and any internal path
        filename = fso.BuildPath(fso.GetSpecialFolder(2 /*TemporaryFolder*/ ), filename);
        BatchFile.BookmarkRoot; //trick to trigger the file extraction from the archive to the system temporary folder
        if (!fso.FileExists(filename)) return "extraction from archive failed"
    }

    EL.nodeTypedValue = GetFileHash(filename, CurrentField.Value);
    return msg + EL.text;
}Check it on the renamer tool with a multiple files selection.
Now move the "EL" inside the function, and check again. 

[Padanges]

The script always returns "extraction from archive failed" for the archives. As I understand - there's a problem with the getFileName() function.
We need to change the line:

Code: [Select]

filename = fso.getFileName(filename); //get archived file name, by removig the archive name and any internal pathto the line:

Code: [Select]

filename = filename.substring(filename.indexOf('>')+1,filename.length); // remove the tag
Thanks for the explanation! So if I get it right, then the line:

Code: [Select]

if (!EL) {is actually:

Code: [Select]

if (typeof EL === 'undefined' || EL === null) {which optimizes the batch process by catching undeleted object.

It is interesting to note that ZIP archives return "(EL created)+HASH" despite "var El;" being either inside function scope, or outside, while RAR archives react to this as all the rest of files. I checked this with BatchInfoEdit tool, not the BatchRename tool though.

[RTT]

The script always returns "extraction from archive failed" for the archives. As I understand - there's a problem with the getFileName() function.
We need to change the line:

Code: [Select]

filename = fso.getFileName(filename); //get archived file name, by removig the archive name and any internal pathto the line:

Code: [Select]

filename = filename.substring(filename.indexOf('>')+1,filename.length); // remove the tag

Indeed! I tested for the worst case (archive with folders), and forgot to test that fso.getFileName in the other, only file name, situation.

Thanks for the explanation! So if I get it right, then the line:

Code: [Select]

if (!EL) {is actually:

Code: [Select]

if (typeof EL === 'undefined' || EL === null) {which optimizes the batch process by catching undeleted object.

Yes, that's the idea.

It is interesting to note that ZIP archives return "(EL created)+HASH" despite "var El;" being either inside function scope, or outside, while RAR archives react to this as all the rest of files. I checked this with BatchInfoEdit tool, not the BatchRename tool though.

Weird! Something to check out.

Here I leave a fixed and enhanced (now also handles file paths longer than 260 characters) CalcHashJS.

Code: [Select]

var fso = new ActiveXObject("Scripting.FileSystemObject");
var EL;

function CalcHashJS() {
    try {
        if (!EL) {
            var MSXML = new ActiveXObject("MSXML2.DOMDocument");
            EL = MSXML.createElement("tmp");
            EL.dataType = "bin.hex";
        }

        var filename = BatchFile.Filename;
        if (filename.indexOf('>') > 0) { //file is archived
            filename = fso.getFileName(filename.substr(filename.indexOf('>') + 1)); //get archived file name, by removig the archive name and any internal path
            filename = fso.BuildPath(fso.GetSpecialFolder(2 /*TemporaryFolder*/ ), filename);
            BatchFile.BookmarkRoot; //trick to trigger the file extraction from the archive to the system temporary folder
            if (!fso.FileExists(filename)) return "extraction from archive failed"
        } else if (filename.length + 1 > 260 /*MAX_PATH*/ ) {
            var f = fso.GetFile('\\\\?\\' + filename);
            filename = f.ShortPath.substr(4);
        }
        EL.nodeTypedValue = GetFileHash(filename, CurrentField.Value);
        return EL.text;
    } catch (err) {
        return err;
    }
}

var stream;

function GetFileStream(filename) {
    if (!stream) {
        stream = new ActiveXObject("ADODB.Stream");
        stream.Type = 1; //adTypeBinary
    }
    stream.Open();
    stream.LoadFromFile(filename);
    return stream
}

function GetFileHash(filename, hashtype) {
    var HasMethodProgID = "";
    switch (hashtype) {
    case BatchFile.Filename:
    case "md5":
        HasMethodProgID = "System.Security.Cryptography.MD5CryptoServiceProvider";
        break;
    case "sha1":
        HasMethodProgID = "System.Security.Cryptography.SHA1Managed";
        break;
    case "sha256":
        HasMethodProgID = "System.Security.Cryptography.SHA256Managed";
        break;
    case "sha384":
        HasMethodProgID = "System.Security.Cryptography.SHA384Managed";
        break;
    case "sha512":
        HasMethodProgID = "System.Security.Cryptography.SHA512Managed";
        break;
    case "ripemd160":
        HasMethodProgID = "System.Security.Cryptography.ripemd160Managed"
        break;
    }
    if (!HasMethodProgID) throw '"' + hashtype + '" is not a valid hash method.';
    try {
        if (GetFileStream(filename)) return ComputeHash(stream, HasMethodProgID)
    } catch (err) {
        throw err
    } finally {
        if (stream && stream.state == 1) stream.Close();
    }
}

function ComputeHash(stream, objectid) {
    var BlockSize = 65536;
    var HashObj = new ActiveXObject(objectid);
    try {
        var nBlocks = parseInt(stream.size / BlockSize);
        var block;
        for (var i = 1; i < nBlocks; i++) {
            block = stream.read(BlockSize);
            HashObj.TransformBlock(block, 0, BlockSize, block, 0);
        }
        var n = stream.size - stream.position;
        block = stream.read(n);
        HashObj.TransformFinalBlock(block, 0, n);
        return HashObj.Hash;
    } finally {
        HashObj.Clear
    }
}


[Padanges]

We have three test case files which fail to get Hash calculated:
1) long names, using "Only DB" option. I suppose only you can see this working if you have fixed this issue which was mentioned in some previous post. So, therefore, I need that update too
2) for files inside archives. I guess it's the same case as previous because I was trying the "Only DB" option. Should we be able to change metadata for archived files? I suppose not.
3) "whatever.pDF" -- please note the letter case for extension. I would argue that extensions should always be case insensitive, for any checks, but haven't found the line which is responsible for this yet. Any hints?

I've also noticed that you have changed block size "var BlockSize = 65536 * 10;" - what was the reason for that? It gives slight calculation speed slowdown.

[RTT]

We have three test case files which fail to get Hash calculated:
1) long names, using "Only DB" option. I suppose only you can see this working if you have fixed this issue which was mentioned in some previous post.

Probably. Even so, I've updated the above last version of the script for some related issues. Try it again.

So, therefore, I need that update too

You will
By the way. Why don't you register in the forum, and continue posting as a guest? Being registered has some advantages, such as being able to receive email notifications when a reply is posted to a topic you are following, attach files, send and receive personal messages, ...

2) for files inside archives. I guess it's the same case as previous because I was trying the "Only DB" option. Should we be able to change metadata for archived files? I suppose not.

DB only when archived file, or file type without metadata edit support, being processed, even if DB Only option not checked.

3) "whatever.pDF" -- please note the letter case for extension. I would argue that extensions should always be case insensitive, for any checks, but haven't found the line which is responsible for this yet. Any hints?

No idea. I made a quick test and worked in that situation. What's the full file path, so I can test here?

I've also noticed that you have changed block size "var BlockSize = 65536 * 10;" - what was the reason for that? It gives slight calculation speed slowdown.

Still figuring out what's the best size. Probably this is dependent on the system (disk, ram, windows version,...)

[Padanges]

Why don't you register in the forum, and continue posting as a guest?

Here we go - now I'm a legit forum member

DB only when archived file, or file type without metadata edit support, being processed, even if DB Only option not checked.

I'm not sure I have understood you correctly: you say that for files which have no metadata support (that includes archives as well) we work only with DB metadata by default, even without "DB Only" option checked/unchecked?

I made a quick test and worked in that situation. What's the full file path, so I can test here?

I've made some other tests and was unable to duplicate this behaviour again but the initial issue persists. Strange. Then, the problem is on my side.

I've updated the above last version of the script for some related issues. Try it again.

Thanks. Just a quick note: 260 is already out of SHORT_PATH bounds therefore you should write "> 259".

One more thing, if I try to RUN/DEGUB the updated code in Script Manager I get a warning "'Sample data' is not a valid hash method.". I suppose this means that in the Script Manager by default you run the line [CalcHashJS](Sample data). Is there a way to run a specific (CurrentField.Value) parameter string from within the Script Manager? Up to this point I used to have to close the Script Manager and change test strings from the "outside".

[RTT]

Why don't you register in the forum, and continue posting as a guest?

Here we go - now I'm a legit forum member

Now with all your previous posts reassigned to your user account.

DB only when archived file, or file type without metadata edit support, being processed, even if DB Only option not checked.

I'm not sure I have understood you correctly: you say that for files which have no metadata support (that includes archives as well) we work only with DB metadata by default, even without "DB Only" option checked/unchecked?

Yes, that's the behavior I'm implementing now, in order to support other file types.

Just a quick note: 260 is already out of SHORT_PATH bounds therefore you should write "> 259".

That's why the above code is filename.length+1>260. Let the compiler optimize the arithmetic. I prefer to read filename+null>MAX_PATH.

One more thing, if I try to RUN/DEGUB the updated code in Script Manager I get a warning "'Sample data' is not a valid hash method.". I suppose this means that in the Script Manager by default you run the line [CalcHashJS](Sample data). Is there a way to run a specific (CurrentField.Value) parameter string from within the Script Manager? Up to this point I used to have to close the Script Manager and change test strings from the "outside".

Adding a parameters input field to the scripts manager, to manually set the value while testing/debugging, is a possibility to fix this. Meanwhile, manually adding a temporary, for debug proposes, CurrentField.Value=YouData line to the code is a easy solution.