0·
4 days agoPython should be able to handle this for you with 100% accuracy at removing duplicated sentences (and headers), with added complexity you could have it even make result files based on unique emails you corresponded with, and it can handle all the various file types and even nested folders if that’s involved.
Probably could do this in 50-100 lines of code depending on complexity. Runtime should be minutes at most.
Learning python to do this would probably take a couple to a few hours if you have any familiarity with coding, bit more if you don’t.
I doubt you’ll find anything for this bespoke purpose built as a utility online, so if you want it you’ll need to make it (or I guess hire someone).
Python is fairly approachable to learn, the 2nd best time to plant a tree is today. Personally, I’d recommend installing Anaconda and using a Jupyter Notebook. For this process, you’d need to learn about:
“pip” to install any libraries/packages you need.
Importing packages
Working with strings
Getting a list of all files (maybe only those of certain types) in a given directory.
For loops
some way to read each file type (pandas can handle txt, not sure about the others but you should be able to literally just Google “Python import <filetype>” to find a package. Import them as strings and concatenate a huge list of them in a for loop.
Pandas dataframes, Parsing strings into new columns, including into distinct lists of strings. The complexity here varies a lot based on what you specifically want to get out of the process.
Exporting dataframe rows to files (better imho if you did it as tabs in a workbook, but it is trickier) in a for loop.
Don’t overwrite objects as you go and keep different logical chunks in their own cells. That’ll make it easier to troubleshoot. Just Google any errors you don’t immediately understand, you’ll find lots of advice online.
Don’t have whatever you make delete any of your original files.
Depending on your feelings about AI, you could probably get one to make something that’ll work after a few tries and troubleshooting by giving it error messages, but aside from the ethical problems, you’d never really know if it got it right, even if it seemed to work, so I would recommend against it.