Quote:

Approximately how many unique lines in total will there be?

If it's no more than (say) 10,000 you can load each line into an array. Search the array for the string, if it's not present add it and write to your new file otherwise ignore it.



Let's say 100 lines max.