Fastest way to compare two files?

[Cobalt]

Anybody out there have a fast way to compare two files?
I have tried using a string(a$=space$(256): Get #5,,a$) to read more bytes of the files and compare them but that was actually slower than comparing the files one LONG integer at a time. fastest I've found at the moment is using INTEGER64s to read 8 bytes at a time to compare larger files but still takes a long time.

[SMcNeill]

OPEN file1$ FOR BINARY AS #1
OPEN file2$ FOR BINARY AS #2
t1$ = SPACE$(LOF(1))
t2$ = SPACE$(LOF(2))
GET #1, 1, t1$
GET #2, 1, t2$

IF t1$ = t2$ THEN It's an EXACT duplicate ELSE It's not...


For long files, I'd simply compare LOF(1) and LOF(2) for a pretest.  If they're not a match then there's no reason to compare the contents.

[RhoSigma]

Steve's way seems to be the fastest just for my logical understanding (not tested), however, if you rather wanna go with a chunky test (eg. INTEGER64) then beside the test "IF a&& = b&&" you also could try "xx&& = aa&& XOR bb&&" and then checking xx& for zero, if it is, then the chunk is identical, if it's not zero, then it's different. No idea, if "=" is faster than "XOR" or vise versa, it's just a thought.

[TempodiBasic]

Hi Cobalt

I agree with Steve's idea, so the first comparison is on the lenght of the file, the second IMHO on the date DD-MM-YYYY and if it is important the hours and minutes of creation, the third step is to compare the internal contents.

so in pseudocode

Code: QB64: [Select]

1.  
2. OPEN file1$ FOR BINARY AS #1
3. OPEN file2$ FOR BINARY AS #2
4. Lenght1 = LOF(1)
5. Lenght2 = LOF(2)
6. IF Lenght1 = Lenght2  THEN
7.        
8.   IF  date_creation_File1 ( DLL_function) = date creation_File2 (DLL function) THEN
9.       IF Hours_creation_File1 (DLL function) = Hours_creation_File2 (DLL function) THEN
10.              EXIT  function/procedure/Subroutine
11.          ELSE
12.           ' here we compare the contents of the two files....
13.             'Following Steve's way
14.  
15.              t1$ = SPACE$(Lenght1)
16.              t2$ = SPACE$(Lenght2)
17.              GET #1, 1, t1$
18.              GET #2, 1, t2$
19.  
20.              IF t1$ = t2$ THEN It's an EXACT duplicate ELSE It's not...
21. ' OR
22. ' RhoSigma's way
23.  
24.             xx&& = aa&& XOR bb&&  ' here you imagine that aa&& and bb&& are the contents of the two file to compare
25.             IF xx&& = 0 THEN It's an EXACT duplicate ELSE It's not...
26.  
27.        END IF
28.   END IF
29. END IF
30.  
31.  

about reference of DLL to get time and date of file creation here a sample from the wiki of QB64 to adapt to this goal

Code: QB64: [Select]

1. File Times
2. ÄÄÄÄÄÄÄÄÄÄ
3.  
4.  
5. CONST GENERIC_READ = -&H80000000
6. CONST GENERIC_WRITE = &H40000000
7. CONST FILE_SHARE_READ = &H1
8. CONST FILE_SHARE_WRITE = &H2
9. CONST OPEN_EXISTING = &H3
10. CONST INVALID_HANDLE_VALUE = -1
11.  
12. DECLARE DYNAMIC LIBRARY "kernel32"
13. FUNCTION CreateFileA%& (BYVAL lpFileName AS _OFFSET, BYVAL dwDesiredAccess AS _UNSIGNED LONG, BYVAL dwShareMode AS _UNSIGNED LONG, BYVAL lpSecurityAttributes AS _OFFSET, BYVAL dwCreationDisposition AS _UNSIGNED LONG, BYVAL dwFlagsAndAttributes AS _UNSIGNED LONG, BYVAL hTemplateFile AS _OFFSET)
14. FUNCTION CloseHandle& (BYVAL hObject AS _OFFSET)
15. FUNCTION GetFileTime& (BYVAL hFile AS _OFFSET, BYVAL lpCreationTime AS _OFFSET, BYVAL lpLastAccessTime AS _OFFSET, BYVAL lpLastWriteTime AS _OFFSET)
16. FUNCTION SetFileTime& (BYVAL hFile AS _OFFSET, BYVAL lpCreationTime AS _OFFSET, BYVAL lpLastAccessTime AS _OFFSET, BYVAL lpLastWriteTime AS _OFFSET)
17. FUNCTION FileTimeToLocalFileTime& (BYVAL lpFileTime AS _OFFSET, BYVAL lpLocalFileTime AS _OFFSET)
18. FUNCTION LocalFileTimeToFileTime& (BYVAL lpLocalFileTime AS _OFFSET, BYVAL lpFileTime AS _OFFSET)
19. FUNCTION FileTimeToSystemTime& (BYVAL lpFileTime AS _OFFSET, BYVAL lpSystemTime AS _OFFSET)
20. FUNCTION SystemTimeToFileTime& (BYVAL lpSystemTime AS _OFFSET, BYVAL lpFileTime AS _OFFSET)
21. FUNCTION GetLastError& ()
22. END DECLARE
23.  
24. TYPE FILETIME
25.   dwLowDateTime AS _UNSIGNED LONG
26.   dwHighDateTime AS _UNSIGNED LONG
27. END TYPE
28.  
29. TYPE SYSTEMTIME
30.   wYear AS _UNSIGNED INTEGER
31.   wMonth AS _UNSIGNED INTEGER
32.   wDayOfWeek AS _UNSIGNED INTEGER
33.   wDay AS _UNSIGNED INTEGER
34.   wHour AS _UNSIGNED INTEGER
35.   wMinute AS _UNSIGNED INTEGER
36.   wSecond AS _UNSIGNED INTEGER
37.   wMilliseconds AS _UNSIGNED INTEGER
38. END TYPE
39.  
40. DIM CreateDate AS FILETIME
41. DIM ModifyDate AS FILETIME
42. DIM AccessDate AS FILETIME
43.  
44. DIM systime AS SYSTEMTIME
45.  
46. DIM FileName AS STRING
47. DIM FileHandle AS _OFFSET
48.  
49. FileName = "readme.txt" + CHR$(0) '<<<<<< Existing file in QB64 folder. Use existing file path!
50.  
51. FileHandle = CreateFileA%&(_OFFSET(FileName), GENERIC_READ, FILE_SHARE_READ OR FILE_SHARE_WRITE, 0, OPEN_EXISTING, 0, 0)
52. IF FileHandle <> INVALID_HANDLE_VALUE THEN
53.   IF GetFileTime&(FileHandle, _OFFSET(CreateDate), _OFFSET(ModifyDate), _OFFSET(AccessDate)) THEN
54.     PRINT HEX$(CreateDate.dwLowDateTime) + HEX$(CreateDate.dwHighDateTime)
55.     PRINT HEX$(ModifyDate.dwLowDateTime) + HEX$(ModifyDate.dwHighDateTime)
56.     PRINT HEX$(AccessDate.dwLowDateTime) + HEX$(AccessDate.dwHighDateTime)
57.     PRINT
58.     IF FileTimeToSystemTime&(_OFFSET(CreateDate), _OFFSET(systime)) THEN
59.       PRINT "Creation time, in GMT, in decimal:"
60.       PRINT "Year:"; systime.wYear
61.       PRINT "Month:"; systime.wMonth, "("; MID$("JanFebMarAprMayJunJulAugSepOctNovDec", (systime.wMonth * 3) - 2, 3); ")"
62.       PRINT "DayOfWeek:"; systime.wDayOfWeek, "("; MID$("SunMonTueWedThuFriSat", (systime.wDayOfWeek * 3) + 1, 3); ")"
63.       PRINT "Day"; systime.wDay
64.       PRINT "Hour"; systime.wHour
65.       PRINT "Minute"; systime.wMinute
66.       PRINT "Second"; systime.wSecond
67.       PRINT "Milliseconds"; systime.wMilliseconds
68.     ELSE
69.       PRINT "FileTimeToSystemTime failed. Error: 0x" + LCASE$(HEX$(GetLastError&))
70.     END IF
71.   ELSE
72.     PRINT "GetFileTime failed. Error: 0x" + LCASE$(HEX$(GetLastError&))
73.   END IF
74.   IF CloseHandle&(FileHandle) = 0 THEN
75.     PRINT "CloseHandle failed. Error: 0x" + LCASE$(HEX$(GetLastError&))
76.     END
77.   END IF
78. ELSE
79.   PRINT "CreateFileA failed. Error: 0x" + LCASE$(HEX$(GetLastError&))
80.   END
81. END IF
82. END  
83.  
84. Code courtesy of Michael Calkins

or here https://docs.microsoft.com/it-it/dotnet/api/system.io.file.getcreationtime?redirectedfrom=MSDN&view=netframework-4.7.2#System_IO_File_GetCreationTime_System_String_

I hope it is useful

[Cobalt]

OPEN file1$ FOR BINARY AS #1
OPEN file2$ FOR BINARY AS #2
t1$ = SPACE$(LOF(1))
t2$ = SPACE$(LOF(2))
GET #1, 1, t1$
GET #2, 1, t2$

IF t1$ = t2$ THEN It's an EXACT duplicate ELSE It's not...


For long files, I'd simply compare LOF(1) and LOF(2) for a pretest.  If they're not a match then there's no reason to compare the contents.

that works up until audio and video files which can be 98% matches but different file sizes and be the exact same video or audio. if someone just trimmed a few seconds off.(side effect of downloading playlists) there might be a few different byte in the beginning or the end from the trim but the bulk is exactly the same. thats why i tried setting a$ to 256,1024 or even lof(#)/100. but it turned out slower than reading long int for long int on files 150MB and up. at the moment I skip anything bigger than 50MB cause compare times can exceed 1200 seconds a cycle. it stores them but just flags them as same file name.

Just struck me as odd that comparing the files with strings at 256 or 1024 bytes at a time was actually taking longer than just reading them 4 bytes at a time with a long integer. within the first 20000 records scanned it finds just over 80000 duplicates mostly from various copies of QB64 I'm sad to say. there seem to often be multiply exact copies of files with the same version of QB64 too.

[SMcNeill]

that works up until audio and video files which can be 98% matches but different file sizes and be the exact same video or audio. if someone just trimmed a few seconds off.(side effect of downloading playlists) there might be a few different byte in the beginning or the end from the trim but the bulk is exactly the same.

Unless you're writing a program to check for percent that matches, you'll always get the same result, whether you check 4 bytes at once, or the whole file contents.

Let's say we have 2 files:

"1234567890"
"1234567890  "       <---- see the extra spaces?

Read those 1 byte, 2  bytes, or all at once, and they're not going to say, "Matches.  File 2 just has a few extra spaces, or File 1 was trimmed somewhat...."

Sounds like what you need is a way to compare percent matches, and then flag them for manual comparison so you can check for yourself if the contents are close enough to be considered "duplicate" by you.

[codeguy]

Why not use instr() if you want to see if some or all of the contents of file 1 is contained in file 2?
If the lengths of the two files match and instr() returns 1, they are exactly the same. If instr() returns a value > 1, file 1 is contained in file 2. If it returns 0, file 1 is not contained in file 2.